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Executive  Summary 


This  is  the  final  report  of  the  Systems  Engineering  Research  Center  (SERC)  research  task  RT-141  that 
finalizes  the  related  tasks  under  RT-48/118.  These  RTs  focused  on  a  Vision  held  by  NAVAIR's  leadership 
to  assess  the  technical  feasibility  of  a  radical  transformation  through  a  more  holistic  model-centric 
engineering  approach.  The  expected  capability  of  such  an  approach  would  enable  mission-based  analysis 
and  engineering  that  reduces  the  typical  time  by  at  least  25  percent  from  what  is  achieved  today  for  large- 
scale  air  vehicle  systems.  The  effort  investigates  the  technical  feasibility  of  moving  to  a  "complete"  model¬ 
centric  lifecycle  and  includes  four  overarching  and  related  tasks  as  shown  in  Figure  1.  These  tasks  include: 

■  Task  1:  Surveying  Industry,  Government  and  Academia  to  understand  the  state-of  the-art  of  a 
holistic  approach  to  model-centric  engineering  ("everything  digital") 

■  Task  2:  Develop  a  common  lexicon  for  things  related  to  models,  including  model  types,  levels, 
uses,  representation,  visualizations,  etc. 

■  Task  3:  Model  the  "Vision,"  but  also  relate  it  to  the  "As  Is"  and  Airworthiness  processes 

■  Task  4:  Integrate  a  Risk  Management  framework  with  the  Vision 


1)  Global  scan  and  classification  of  holistic 
state-of-the-art  MBSE 


Use  discussion  framework  to  survey 
government,  industry  and  academia 
Quantify,  link 
and  trace  realized 
modeling 
capabilities 
to  Vision  (task  3) 


2)  Develop  Common  Lexicon  for  Model 
Levels,  Types,  Uses,  and  Representations 
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Address  two  classes  of 
risk: 

Airworthiness  and 
Safety 

Program  Execution 


3)  Model  the  Vision  of  Everything  Done  with 
Models  and  Relate  to  “As  Is”  process 


4)  Fully  integrate  model-driven  Risk 
Management  and  Decision  Making 


Figure  1.  Four  Tasks  to  Assess  Technical  Feasibility  of  "Doing  Everything  with  Models" 


There  has  been  considerable  emphasis  on  understanding  the  state-of-the-art  through  discussions  with 
industry,  government  and  academia.  We  have  conducted  over  29  discussions,  including  21  on  site,  and 
15  working  sessions,  as  well  as  several  follow-up  discussions  on  some  of  the  identified  challenge  areas. 
We  did  not  do  a  survey,  but  rather  had  open-ended  discussions.  We  asked  the  meeting  coordinators  to 
in  general: 
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Tell  us  about  the  most  advanced  and  holistic  approach  to  model-centric  engineering  you  use  or  have 
seen  used. 

The  spectrum  of  information  was  very  broad;  there  really  is  no  good  way  to  make  a  comparison.  In 
addition,  we  had  proprietary  information  agreements  with  most  industry  organizations.  The  objective 
was  not  to  single  out  any  specific  organization,  therefore,  we  will  summarize,  in  the  aggregate,  what  we 
heard  in  this  report  as  it  relates  to  the  NAVAIR  research  objective. 

Our  research  suggests  that  model-centric  engineering  is  in  use  and  adoption  seems  to  be  accelerating. 
Model-centric  engineering  can  be  characterized  as  an  overarching  digital  engineering  approach  that 
integrates  different  model  types  with  simulations,  surrogates,  systems  and  components  at  different  levels 
of  abstraction  and  fidelity  across  disciplines  throughout  the  lifecycle.  Industry  is  trending  towards  more 
integration  of  computational  capabilities,  models,  software,  hardware,  platforms,  and  humans-in-the- 
loop.  The  integrated  perspectives  provide  cross-domain  views  for  rapid  system  level  analysis  allowing 
engineers  from  various  disciplines  using  dynamic  models  and  surrogates  to  support  continuous  and  often 
virtual  verification  and  validation  for  tradespace  decisions  in  the  face  of  changing  mission  needs. 

Enabling  digital  technologies  are  changing  how  organizations  are  conceptualizing,  architecting,  designing, 
developing,  producing,  and  sustaining  systems  and  systems  of  systems  (SoS).  Some  use  model  and 
simulation  environments  for  customer  engagements,  as  well  as  design  engineering  analyses  and  review 
sessions.  While  they  do  use  commercial  technologies,  most  have  been  innovating  and  have  developed  a 
significant  amount  of  enabling  technology  -  some  call  it  their  "secret  sauce."  The  research  findings  and 
recommendations  are  based  on  seeing  demonstrations  and  evidence  of  cross-cutting  technologies  and 
methods.  Demonstrations  have  included  mission-level  simulations  that  are  being  integrated  with  system 
simulation,  digital  assets  and  aircraft  products  providing  cloud-like  services  enabled  by  the  industrial 
Internet.  There  have  been  demonstrations  of  ID,  2D,  and  3D  modeling  and  simulations  with  a  wide  array 
of  solvers  and  visualization  capabilities.  We  have  also  been  in  an  immersive  Cave  Automated  Virtual 
Environment.  We  have  seen  the  results  of  platform-based  approaches  directly  focused  on  speed-to- 
market,  and  more. 

The  analysis  of  captured  evidence  in  this  research  suggests  that  there  is  a  transition  from  model-based 
engineering  to  model-centric  engineering.  The  advances  and  availability  of  high  performance  computing, 
capabilities  to  provide  cross-domain  and  multi-physics  model  integration,  and  methods  and  tools  to 
assess  model  integrity  will  support  the  need  for  reducing  the  time  to  deliver  system  capabilities.  Even 
sociotechnical  computing  is  enabling  new  ways  to  access  and  more  transparently  collaborate  and  share 
information,  and  it  can  be  a  key  contributor  to  a  radical  transformation  to  model-centric  engineering. 

Findings 

The  findings  conveyed  to  NAVAIR  leadership  definitely  indicate  that  it  is  technically  feasible  to  transform 
systems  engineering  at  NAVAIR  similar  to  the  transformation  seen  across  large  organizations  in 
aerospace,  automotive,  and  government.  This  transformation  increases  the  likelihood  of  achieving  at 
least  25  percent  reduction  in  acquisition.  A  summary  of  the  data  analysis  is  presented  in  a  traceability 
matrix  that  captured  21  topic-discussion  areas  summarized  in  this  report.  The  matrix  also  provided 
evidence  of  traceability  to  different  instances  of  organizational  use  and  their  possible 
impacts/relationships  on  characteristics,  such  as:  performance,  integrity,  affordability,  risk, 
methodologies,  and  within  a  single  source  of  technical  truth1. 


1  Our  sponsor  uses  the  term  single  source  of  technical  truth;  others  have  used  the  phrases  such  as:  single  source  of 
truth,  single  authoritative  representation  of  the  system.  Any  of  these  terms  apply  to  this  concept. 
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A  rule  of  thumb  is  that  the  effort/time  to  get  from  Milestone  A  to  Critical  Design  Review  (CDR)  is  about 
30  percent  of  the  total  time,  where  the  time  from  CDR  to  Initial  Operating  Capability  (IOC)  is  about  70 
percent  of  the  total  time.  With  some  of  the  new  approaches  to  produce  digital  information,  which 
considers  modeling  and  simulation  analysis  of  manufacturability  prior  to  CDR,  the  digital  information  at 
CDR  could  significantly  reduce  the  70  percent  effort  from  CDR  to  IOC,  which  also  builds  to  the  argument 
for  being  able  to  reduce  the  acquisition  time  by  25  percent  with  MCE. 

The  feasibility  of  Systems  Engineering  transformation  through  MCE  has  three  key  critical  technical  items: 
1)  cross-domain  and  multi-physics  model  integration,  2)  ensuring  model  integrity  (trust  in  the  model 
predictions),  and  3)  high  performance  computing,  which  is  an  enabler  for  1  and  2,  but  critical  due  to  the 
scale  and  complexity  of  next  generation  systems. 

There  are  a  number  of  examples  that  span  various  domains  across  aerospace,  automotive,  and  involving 
commercial,  government  and  academic  organizations.  Many  have  lessons  learned  and  examples  covering 
a  number  of  themes  spanning  technologies,  methods,  and  usage  at  various  stages  of  the  lifecycle,  even 
taking  into  consideration  constraints  for  manufacturability  in  design-space  exploration.  Therefore  MCE  is 
not  necessarily  the  catalyst;  rather  it  is  enabled  by  approaches  that  support  data-driven  decision-making 
that  will  subsume  processes  through: 

■  Single  Source  of  Technical  Truth  (SSTT)  -  one  source  of  information 

■  Views  and  viewpoints  for  the  multidisciplinary  stakeholders  into  the  SSTT 

■  Multidisciplinary  Design,  Analysis  and  Optimization  (MDAO)  in  both  tradespace  exploration  and 
analysis  of  the  problem  and  design  space 

■  Workflow  orchestration  -  by  having  the  data  dependencies  being  semantically  linked  within  the 
SSTT 

■  Enabled  by  High  Performance  Computing  (HPC) 

Recommendation 

NAVAIR  senior  leadership  confirmed  that  the  research  finding  and  analysis  have  validated  their  vision 
hypothesis  stated  at  the  System  Engineering  Transformation  kickoff  meeting  of  RT-48.  They  conclude  that 
NAVAIR  must  move  quickly  to  keep  pace  with  the  other  organizations  that  have  adopted  MCE  and  who 
continue  to  evolve  at  an  accelerating  pace  enabled  by  the  advances  in  technologies  and  improved 
methods.  NAVAIR  must  also  transform  in  order  to  continue  to  perform  effective  oversight  of  weapon 
system  development  by  primes  that  are  using  modern  modeling  methods  for  system  development.  The 
risks  of  not  moving  forward  include  making  acquisition  decisions  with  progressively  less  technical-truth 
insight  and  the  proliferation  of  disparate,  redundant  and  stove-piped  data  and  models,  and  lacking  MCE 
capabilities  and  knowledge  needed  to  understand  an  increasingly  complex  problem  and  design  space. 

The  path  forward  has  challenges  but  also  many  opportunities,  both  technical  and  sociotechnical.  It  must 
include  a  modeling  framework  with  HPC  that  enables  SSTT,  integration  of  multi-domain  and  multi-physics 
models,  and  provides  for  a  method  for  model  integrity.  The  modeling  and  infrastructure  for  a  digital 
engineering  environment  is  a  critical  step  to  enable  a  SSTT.  While  there  are  literally  thousands  of  tools, 
they  are  often  federated  and  there  is  no  one  single  solution  that  can  be  purchased.  Every  organization 
providing  inputs  to  this  research  has  had  to  architect  and  engineer  their  model-centric  engineering 
environment,  most  have  selected  commercial  tools  and  have  developed  the  integrating  fabric  between 
the  different  tools,  models,  and  data.  This  approach  often  uniquely  positions  them  with  some  advantages 
among  the  rest.  Some  organizations  have  encoded  historical  knowledge  in  reference  models,  model 
patterns  to  embed  methodological  guidance  to  support  continuous  orchestration  of  analysis  through  new 
modeling  metrics,  automated  workflow,  and  more.  The  items  to  investigate  further  include  but  are  not 
limited  to: 
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■  Cross-domain  integration  of  models  to  address  the  heterogeneity  of  the  various  tools  and 
environments 

■  Model  integrity  to  ensure  trust  in  the  model  predictions  by  understanding  and  quantifying 
margins  and  uncertainty 

■  Modeling  methodologies  that  can  embed  demonstrated  best  practices  and  provide 
computational  technologies  for  real-time  training  within  digital  engineering  environments 

■  Multidisciplinary  System  Engineering  transformation  roadmap  that  looks  across: 
o  Technologies  and  their  evolution 

o  How  people  interact  through  digitally  enabled  technologies  and  new  needed  competencies 
o  How  methodologies  enabled  by  technologies  change  and  subsume  processes 
o  How  acquisition  organizations  and  industry  operate  in  a  digital  engineering  environment 
throughout  the  phases  of  the  lifecycle  (including  operations  and  sustainment) 
o  Governance  within  this  new  digital  and  continually  adapting  environment 

This  report  aggregates  information  contained  in  the  final  technical  reports  of  RT-48  and  RT-118  so  that 
readers  can  get  the  key  information  from  this  report.  The  report  is  structured  so  that  the  key  findings  and 
next  steps  are  described  in  the  first  section.  The  report  then  provides  updated  clarification  on  the  scope 
given  by  our  NAVAIR  sponsor.  Part  II  section  provide  additional  detail  to  summarize  the  efforts  that  are 
aligned  with  tasks  1  through  4. 
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Introduction 


In  2013,  the  Naval  Air  Systems  Command  (NAVAIR)  at  the  Naval  Air  Station,  Patuxent  River,  Maryland 
initiated  a  research  task  (RT-48)  to  assess  the  technical  feasibility  of  creating/leveraging  a  more  holistic 
Model-Based  Systems  Engineering  (MBSE)  approach  to  support  mission-based  analysis  and  engineering 
in  order  to  achieve  a  25  percent  reduction  in  development  time  from  that  of  the  traditional  large-scale 
air  vehicle  weapon  systems.  The  research  need  was  focused  on  the  evaluation  of  emerging  system  design 
through  computer  (digital)  models.  The  first  phase  of  the  effort  under  RT-48  created  a  strategy  and  began 
collecting  and  structuring  evidence  to  assess  the  technical  feasibility  of  moving  to  a  "complete"  model- 
driven  lifecycle.  The  second  phase  conducted  under  RT-118  involved  an  extensive  outreach  to  understand 
the  state-of-the-art  in  using  models.  The  third  phase  under  RT-141  conducted  additional  discussions  on 
leading-edge  approaches  and  correlated  the  analysis  to  provide  evidence  that  traced  to  organizational 
examples  needed  to  finalize  the  finding  and  recommendations. 

A  goal  is  to  leverage  virtual  designs  that  integrate  with  existing  systems  data  and  simulations,  as  well  as 
surrogates  at  varying  levels  of  refinement  and  fidelity  to  support  a  more  continuous  approach  to  mission 
and  systems  analysis  and  design  refinement.  What  emerged  as  Model-Centric  Engineering  (MCE)  can 
equally  well  be  characterized  as  Digital  Engineering.  This  broader  view  of  the  use  of  models  has  moved 
our  team  to  use  the  term  model-centric  engineering.  MCE  can  be  characterized  as  an  overarching  digital 
approach  for  integrating  different  model  types  with  simulations,  surrogates,  systems  and  components  at 
different  levels  of  abstraction  and  fidelity,  including  software-,  hardware-,  platform-,  and  human-in-the- 
loop  across  disciplines  throughout  the  lifecycle.  The  expanse  of  the  organizational  discussions  even 
included  modeling  and  simulation  to  assess  the  manufacturability  of  a  design  during  tradespace  analysis 
of  the  designs,  and  therefore  we  use  the  term  MCE  in  the  broadest  way  in  this  report. 

The  larger  context  of  the  NAVAIR  mission  seeks  a  radical  transformation  in  the  way  they  operate  within 
NAVAIR  and  with  industry  through  MCE.  The  Vision  of  NAVAIR  is  to  establish  an  environment  to  evaluate 
the  emerging  system  design  through  computer  models  and  demonstrate  system  compliance  to  user 
performance  and  design  integrity  requirements,  while  managing  airworthiness  risks.  It  is  anticipated  that 
this  model-centric  approach  can  streamline  and  radically  transform  the  traditional  document-centric 
process  that  decomposes  requirements  and  their  subsequent  integrated  analysis,  which  is  currently 
aligned  with  the  Department  of  Defense  (DoD)  systems  engineering  V-model  (i.e.,  the  "V").  By  providing 
more  tightly  coupled  and  dynamic  linkages  between  the  two  sides  of  the  traditional  "V,"  more  efficient 
and  focused  requirements  decomposition  would  eliminate  thousands  of  pages  of  documentation 
delivered  via  contract  data  requirements  that  now  substitute  for  directly  invoking,  manipulating,  and 
examining  the  design  through  computer-based  models  and  digital  artifacts. 


Objective 

The  objectives  characterized  by  the  sponsor  at  the  RT-48  kickoff  meeting  were  re-stated  at  the  out 
briefing  to  senior  leadership  at  NAVAIR  to  ensure  that  the  research  covered  the  key  objectives;  those 
objectives  included: 

■  Include  both  models  to  assess  "performance"  and  models  for  assessing  "integrity"  such  as: 
o  Performance:  aero,  propulsion,  sensors,  etc. 

o  Integrity:  Finite  Element  Analysis  (FEA),  Computation  Fluid  Dynamics  (CFD),  reliability,  etc.  - 
can  we  build  it,  can  we  trust  it 

o  A  stated  challenge  was:  how  can  "integrity"  be  accomplished  when  the  current  situation 
involves  federations  of  models  that  are  not  integrated? 
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■  Continuous  hierarchical  and  vertical  flow  enabled  by  models  and  iterative  refinement  through 
tradespace  analysis,  concept  engineering,  and  architecture  and  design  analysis 

■  Integrated  mission  area  model  analysis  supported  by  a  modeling  infrastructure  starting  at  pre¬ 
milestone  A 

■  Continuous  and  iterative  refinement  of  digital  (model)  versions  representing  digital 
characterizations  of  the  information  for  Preliminary  Design  Review  (PDR) 

o  Analyzed  with  integrated  performance  and  integrity  capabilities 

o  Integrated  with  bidirectional  vertical  flows  between  mission-level  modeling  and  assessment, 
and  the  system(s)  of  interest  (Program  of  Record) 
o  Refinement  of  lower-fidelity  PDR  models  to  higher-fidelity  Critical  Design  Review  (CDR) 
models  characterizing  the  incrementally  refined  Concept  of  Operations  (CONOPS), 
architecture,  and  detailed  design  and  constraints 
o  Models-to-manufacturability  and  models-to-training  were  desired 

Therefore,  there  are  three  overarching  research  questions: 

1.  Is  it  technically  feasible  to  use  model-centric  engineering  in  order  to  achieve  at  least  a  25  percent 
reduction  in  the  time  it  takes  to  deliver  a  large-scale  air  vehicle  weapon  system? 

There  is  a  corollary  to  first  question: 

2.  If  we  are  going  to  rely  more  heavily  on  model-centric  engineering,  with  an  increasing  use  of  modeling 
and  simulations,  how  do  we  know  that  models/simulations  used  to  assess  "performance"  have  the 
needed  "integrity"  to  ensure  that  the  performance  predictions  are  accurate? 

While  model-centricity  may  enable  improved  automation  and  greater  efficiency,  NAVAIR  seeks  its  use  in 
an  improved  operational  paradigm,  and  therefore  the  third  questions  is: 

3.  Can  we  radically  transform  the  way  that  NAVAIR  and  all  contributing  stakeholders  operate  in 
conceptualizing,  architecting,  designing,  developing,  producing,  and  sustaining  systems  and  SoS? 

Our  sponsor  in  the  original  kickoff  briefing  stated: 

■  "Blow  up"  the  current  "Newtonian"  approach  and  move  to  a  "Quan 

■  turn"  approach  that  recognizes  and  capitalizes  on  current  and  emerging  trends  and  enabling 
technologies 

It  is  acknowledged  that  there  are  many  possible  hurdles  beyond  technical  feasibility  (e.g.,  organizational 
adoption,  training,  usability,  etc.),  but  they  have  in  general  been  reduced  in  priority  for  this  phase  of  the 
effort.  The  path  forward  includes  a  road  map  to  factor  in  some  of  these  other  considerations. 

The  focus  of  the  research  task  is  scoped  at  the  system  level,  often  characterized  as  the  Program  of  Record 
(POR)  plus  weapons,  for  an  air  vehicle  system,  although  the  next  steps  are  directed  to  include  Systems  of 
Systems  (SoS).  The  timeframe  for  the  technical  feasibility  is  scoped  at  10  years.  This  would  reflect  on  the 
"Vision"  concept,  but  also  the  operational  aspects  of  government  organizations  like  NAVAIR  interacting 
in  a  more  continuous  type  of  collaboration  with  industry  stakeholders. 


Scope 

Given  the  objectives  above,  we  were  directed  to  scope  the  effort  to  focus  on  the  front-end  of  the  lifecycle 
from  pre-milestone  A  to  critical  design  review  (CDR).  This  is  typically  considered  the  front  half  of  the  "V" 
model.  However,  as  is  discussed  later  in  this  report,  many  of  our  meeting  discussions  went  well  beyond 


6 


this  scope.  We  do  document  most  of  these  potential  out-of-scope  ideas  as  they  may  play  a  role  in  a  radical 
transformation. 


Organization  of  Document 

Section  1  provides  an  overview  of  the  research  objectives,  scope  and  organization  of  this  report. 

Section  2  provides  the  summary  of  our  efforts,  findings,  analysis  and  recommendations  including  key 
aspects  from  RT-48/118. 

In  order  to  be  comprehensive,  we  are  including  discussions  of  the  four  tasks  in  a  manner  consistent  with 
the  final  technical  report  from  RT-48/118  [20]  [22], 

Section  3  describes  the  approach  for  having  discussions  with  commercial,  government,  and  academic 
organizations  in  order  to  assess  the  most  holistic  state-of-the-art  use  or  vision  for  model-centric 
engineering. 

Section  4  describes  the  approach  for  developing  a  model  lexicon  to  characterize  such  things  as:  model 
levels,  model  types,  model  uses,  representations,  and  other  categories  related  to  "models." 

Section  5  discusses  the  scope  and  concept  of  the  Vision  model  and  its  relationship  to  the  "As  Is"  artifacts 
and  process  that  are  currently  in  place  for  developing  NAVAIR  air  vehicle  systems;  this  also  includes  the 
airworthiness  process.  This  section  provides  some  context  about  the  scope  of  the  effort  and  the 
relationship  to  the  NAVAIR  mission-level  modeling  and  simulation. 

Section  6  discusses  a  framework  for  risk  identification  and  management  that  is  primarily  focused  on  how 
airworthiness  and  safety  risk  can  be  integrated  in  the  Vision  model.  This  section  discusses  classes  of  risks 
that  we  need  to  deal  with  in  the  context  of  model-centric  engineering,  tools  and  methods  for 
quantification  of  margins  and  uncertainty,  model  validation,  verification,  accreditation,  and  simulation 
qualification. 

Section  7  provides  some  conclusions  with  a  brief  summary  of  the  planned  next  steps. 

There  is  additional  backup  information,  including  the  factor  definitions  associated  with  the  discussion 
collection  instrument,  a  list  of  acronyms  and  abbreviations  following  the  conclusion. 
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Research  Summary 


This  section  provides  a  summary  of  the  findings,  analysis,  examples  and  recommendations  of  this 
research  as  the  final  deliverable  for  the  four  tasks  discussed  under  RTs-48/118/141.  This  section  presents 
the  following  information: 

■  Model-centric  engineering  terminology 

■  Model  lexicon  status 

■  Change  nature  of  system  engineering  needed  for  Cyber  Physical  Systems 

■  15  perspectives  identified  as  themes  from  visits  to,  and  discussions  with  industry,  government, 
and  academia  to  seek  out  the  most  advanced  holistic  uses  of  model-centric  engineering 

■  Discussion  about  some  challenges  areas 

■  Summary  and  next  steps 


Modeling  Terminology  and  Model  Lexicon  Status 

Modeling  terminology  can  be  confusing,  and  we  created  a  model  lexicon  (Task  2).  However,  a  simple 
definition  is  not  always  adequate  as  there  are  many  overlapping  definitions.  Some  of  the  terms  are 
overloaded,  and  some  with  multiple  definitions.  The  Lexicon  may  also  need  to  be  tailored  to  the  needs  of 
the  organization  and  account  for  different  definitions  within  different  parts  of  the  organization.  While  we 
did  give  some  references  and  example  uses  in  our  lexicon,  they  do  not  necessarily  completely  convey  the 
broad  concepts  such  as  model-centric  engineering. 

Status  Task  2:  we  have  captured  over  750  named  lexicon  items  related  to  the  term  "model,"  including 
levels,  types,  uses,  representations,  standards,  etc.  There  were  many  contributors  to  the  lexicon.  It 
has  been  reviewed  several  times,  and  updated  based  on  the  comments  of  the  reviewers.  We  have 
delivered  these  model-lexicon  artifacts  to  NAVAIR  for  them  to  post  internally. 

The  directive  when  we  started  this  research  task  was  to  investigate  the  most  advanced  approaches  to 
Model-Based  Systems  Engineering  (MBSE).  However,  during  our  organizational  discussions  some  of  the 
people  we  interacted  with  thought  this  to  be  Model-Based  Engineering  (MBE),  others  used  the  term 
Model-Centric  Engineering  [111],  Integrated  Model-Centric  Engineering  [8],  Interactive  Model-Centric 
Engineering  [101],  Model-Driven  Development,  Model-Driven  Engineering  (MDE)  [13],  and  even  Model- 
Based  Enterprise  [81],  which  brings  in  more  focus  on  manufacturability.  The  concept  characterized  as 
Digital  Thread2  envisions  a  frameworks  that  merges  physics-based  models  generated  by  the  discipline 
engineers  during  the  detailed  design  process  with  MBSE's  conceptual  and  top-level  architectural  models, 
resulting  in  a  single  authoritative  representation  of  the  system  [121].  The  words  Digital  Engineering  have 
been  used,  because  the  nature  of  systems  engineering  is  expanding  to  include  more  rigorous  uses  of 
models  and  cross-domain  integration. 


2  Digital  Thread:  "An  extensible,  configurable  and  component  enterprise-level  analytical  framework  that  seamlessly 
expedites  the  controlled  interplay  of  authoritative  technical  data,  software,  information,  and  knowledge  in  the 
enterprise  data-information-knowledge  systems,  based  on  the  Digital  System  Model  template,  to  inform  decision 
makers  throughout  a  system's  life  cycle  by  providing  the  capability  to  access,  integrate  and  transform  disparate 
data  into  actionable  information." 
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Changing  Nature  of  Engineering  and  the  Impact  on  Systems  Engineering 


Systems  have  been  categorized  as:  Physical,  Computational,  and  Cyber  Physical  Systems  (CPS)  [116]. 
While  the  research  did  not  attempt  to  focus  on  a  particular  type  of  system,  invariably  the  discussions 
involved  CPS.  The  phrase  "cyber-physical  systems"  was  coined  by  Helen  Gill  [55]  and  defined  as  physical, 
biological,  and  engineered  systems  whose  operations  are  integrated,  monitored,  and/or  controlled  by  a 
computational  core.  Components  are  networked  at  every  scale.  Computing  is  deeply  embedded  into 
every  physical  component,  possibly  even  into  materials.  The  computational  core  is  an  embedded  system, 
usually  demands  real-time  response,  and  is  most  often  distributed."  CPSs  are  integrations  of 
computation,  networking,  physical  processes  and  often  involving  humans-in-the-loop.  Many  advanced 
systems  have  computational  intelligence  enabling  self-adaptation  and  flexible  autonomy. 

CPS  such  as  aircraft,  automobiles,  robotics,  and  space  systems  involve  multi-discipline,  multi-physics,  and 
multi-vendors  during  conceptualization  of  the  problem  space,  architecting  the  design  space,  generation 
and  composability  during  implementation,  and  manufacturability  of  these  systems.  The  magnitude  of  the 
tradespace  is  near  infinite  and  the  document-centric  tools  and  methods  for  engineering  systems  today 
cannot  address  even  complicated  systems,  much  less  complex  systems.  The  impact  of  decisions  regarding 
one  design  parameter  on  other  design  parameters  is  difficult,  at  best,  to  determine  and  can  hamper 
effective  trade  space  analysis.  MCE  provides  some  of  the  needed  support  by  integrating  different  model 
types  and  tools  for  the  simulation  and  analysis  of  systems  and  components  at  various  levels  of  abstraction 
and  fidelity  across  disciplines  and  throughout  the  lifecycle  [120], 

Some  of  the  sponsor  leadership  sees  less  complex  technology  development  outside  of  systems 
engineering  like  the  mobile  app  phenomenon,  and  they  have  asked,  "Why  is  this  (MCE  on  the  scale  of 
large  scale  aircraft,  e.g.,  the  Joint  Strike  Fighter)  so  complicated?"  Engineers  and  their  customers  live  in  a 
world  full  of  consumer  technology  with  powerful  computing  ability  available  in  almost  every  aspect  of 
their  lives  (vehicles,  health  care,  smart  homes,  smart  phones,  etc.).  This  availability  of  connectedness  in 
everyday  life  that  streamlines  personal  needs  creates  desired  environments  within  the  workplace. 
Electronic  records,  calendars,  forms,  databases  that  are  easy  to  search  and  link  in  commercial  cloud 
environments  are  desired  by  engineers  over  paper  documentation  that  is  viewed  as  hard  to  locate  and 
link  to  other  artifacts.  In  daily  life  if  a  person  decides  they  would  like  to  be  able  to  link,  monitor,  or  find 
something  they  simply  go  the  application  store  find  an  "app"  and  their  need  is  met.  The  problem  is  that 
these  are  primarily  in  the  computational  domain,  and  not  cross-domain  or  multi-physics  like  CPS. 

Customers  and  engineers  desire  a  more  seamless  flow  of  system  development  without  duplicate  or 
overlapping  effort.  System  developers  are  looking  for  ways  to  generate  digital  threads  extracted  from  the 
single  source  of  technical  truth.  These  digital  threads  would  allow  system  developers  to  trace  Concept  of 
Operations  (CONOPS)  through  tradespace  analyses,  to  requirements  all  the  way  through  the 
development  process  and  reduce  the  number  of  regeneration  and  duplication  of  material  through  out 
the  development  process.  A  single  source  of  truth  could  allow  for  easier  recognition  of  impacts  due  to 
design  and  requirement  changes,  better  indicators  of  program  status  (both  technical  and  management), 
and  earlier  recognition  of  design/requirement  anomalies  or  issues. 

While  MCE  is  a  newer  concept  with  limited  measurable  data,  the  state  of  MBSE  has  been  tracked  for 
some  time.  The  perceived  value  of  MBSE  from  practitioners  in  a  2015  study  showed  a  4.26  rating  on  a 
scale  from  one  to  five  (with  five  being  high  value)  [37],  This  study  showed  also  overtime  this  perceived 
value  has  increased  from  4.05  in  2012  to  the  4.26  in  2015.  The  study  did  not  show  why  the  perceived 
value  has  increased  but  it  is  theorized  it  could  relate  to  better  tools  and  more  people  able  to  execute 
MBSE  well  due  to  continued  training  and  practicing.  While  MBSE  is  a  part  of  the  MCE  it  does  not 
encompass  the  full  idea  and  enabling  technologies  of  MCE.  Traditionally,  MBSE  has  had  limited  use  of 
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static  modeling  tools  often  focused  mostly  on  architecture  structure.  It  is  only  mentioned  here  as  an 
indicator  of  the  value  that  MBSE  and  MCE  can  bring  to  engineering  when  we  expand  the  concept  based 
on  the  evidence  demonstrated  in  our  organizational  visits  with  Industry,  Government  and  Academia. 


Current  State  of  Model  Centric  Engineering 

There  is  no  formalized  definition  for  MCE,  which  this  research  has  characterized  as  an  overarching  digital 
approach  for  integrating  different  model  types  with  simulations,  surrogates,  systems,  software, 
hardware  and  components  at  different  levels  of  abstraction  and  fidelity  across  disciplines  throughout 
the  lifecycle,  including  considerations  for  manufacturability.  Our  organizational  discussions  found  that 
MCE  is  in  use  and  adoption  is  evolving. 

We  use  the  term  MCE  in  the  most  broad  way,  and  see  it  as  an  overarching  approach  to  the  more  general 
digital  engineering  because  there  are  no  one-size  fits  all  formalized  methods,  language,  or  tools;  each 
organization  and  even  different  programs  inside  the  same  organization  implement  MCE  differently  at  this 
time.  Critical  to  our  characterization  is  the  capability  to  achieve  multi-domain  integrated  analysis  enabling 
continuous  virtual  V&V  where  the  interfaces  are  formalized:  1)  structurally,  2)  behaviorally,  and  3) 
temporally  in  order  to  use  surrogates  and  simulations  in  a  semantically  precise  way. 

To  illustrate  the  breadth  of  the  applicable  topics,  we  provide  examples  discussed  in  our  organizational 
visits.  We  emphasize  methods,  models  and  tools  in  use.  In  our  data  collection,  we  used  a  set  of  factors  to 
collect  evidence.  If  we  did  not  see  evidence  related  to  a  factor,  then  that  organization  did  not  "get  credit" 
for  it,  even  if  we  thought  they  probably  did  use  some  aspect  of  a  MCE  capability  related  to  that  factor. 


Operational  Perspective  of  Model-Centric  Integration 

Model-centric  views  provide  a  means  to  integrate  different  model  types  with  simulations,  surrogates, 
systems  and  components  at  different  levels  of  abstraction  and  fidelity.  Figure  2  provides  an  example 
documented  in  a  case  study  that  was  published  in  2008  [57],  The  results  from  the  report  suggest  that  the 
MCE  concepts  have  been  applied  under  various  operational  scenarios  dating  back  at  least  eight  years. 
This  effort  brought  together  different  subject  matter  experts,  who  all  learned  how  to  communicate  using 
the  System  Modeling  Language  (SysML)  [93],  Hidden  behind  the  scenes,  there  was  manually  created  code 
to  integrate  the  levels  and  views  with  simulations,  middleware,  and  legacy  components,  which  evolved 
into  a  working  system.  This  reflects  on  the  fact  that  software  skills  may  be  required  to  assemble  model¬ 
centric  simulations  for  analysis  until  we  improve  the  integration  and  interoperability  of  models  across  the 
domains.  However,  there  has  been  much  advancement  in  tools,  methods  and  computational  power  over 
the  past  eight  years.  Our  follow-up  research  [24]  [74]  and  discussions  under  RT-141  suggests  that  data 
interoperability  will  play  a  larger  roll  enabling  cross-domain  integration,  without  doing  tool-to-tool 
integration. 
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Visual 


Figure  2.  Model  Centric  Provides  Digital  Integration  Between  Views 


Virtual  Verification  and  Validation  Continuum 

Extending  the  previous  example  and  relating  it  to  a  scenario  of  moving  through  the  lifecycle  phases,  our 
team  provided  another  representation  that  was  included  in  the  RT-48  final  technical  report  [20]  that 
extends  this  concept  and  relates  to  the  concept  of  iterative  and  successive  refinement  across  the  lifecycle 
phases.  This  example  is  also  abstract,  but  reflects  on  a  NAVAIR  objective,  which  is  to  continuously  "cross 
the  virtual  V"  early  in  the  lifecycle  in  order  to  better  ensure  that  the  system  design  meets  the  SoS  mission 
needs,  as  well  as  identify  requirements/design  issues  earlier  in  development  cycle  to  reduce  cost/impact 
of  corrective  actions  before  coding  or  "cutting  metal,"  and  ultimately  to  collapse  the  timeline.  The 
evidence  from  the  DARPA  META  projects  provides  support  for  this  concept  [6]  [7], 

Consider  the  following  scenario  using  the  typical  release  phase  reviews  as  the  time  points  to  represent  a 
notional  timeline  moving  from  left  to  right  (e.g.,  System  Requirements  Review  (SRR),  System  Functional 
Review  (SFR),  Preliminary  Design  Review  (PDR),  Critical  Design  Review  (CDR)).  In  a  MCE  world  the  models 
at  SRR  would  reflect  on  the  new  high-level  aircraft  needs/capabilities,  as  conceptually  rendered  in  Figure 
3;  there  would  likely  be  a  strong  relationship  between  these  new  operational  capabilities  and  the  mission 
needs.  These  models  would  need  to  be  sufficiently  "rich"  that  we  could  computationally  connect  them  to 
other  surrogates,  such  as  software  components  (new/legacy),  hardware  and  physical  surrogates  (e.g., 
previous  generation  aircraft).  We  ideally  want  to  have  some  type  of  dynamic  operational  capability 
operating  from  the  very  beginning  of  the  effort  (all  digitally).  As  we  transition  through  the  lifecycle  phases 
SRR,  SFR,  CDR,  and  PDR,  we  would  use  a  similar  process  on  lower-level  models  and  components  that 
provide  increasing  levels  of  fidelity  that  is  moving  us  from  the  analysis  of  the  problem  and  aircraft  needs 
and  closer  to  the  actual  system  as  the  decisions  for  the  physical  design  are  defined  and  refined.  In 
addition,  the  tradeoffs  and  impacts  of  these  decisions  across  the  system  and  SoS  are  better  understood. 
We  begin  to  focus  on  detailed  functional  and  timing  behavior,  with  models  that  predict  performance 
characteristics  and  begin  to  clarify  the  margins  and  uncertainty;  we  would  continue  the  transition  from 
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the  use  of  surrogates  to  versions  of  the  implemented  design.  As  we  continue  to  move  through  the 
acquisition  phases  to  CDR,  especially  for  5th  generation  air  vehicle  systems,  we  will  have  much  more 
software  than  ever  before,  including  software  that  connects  models  with  the  simulations,  surrogates, 
components  and  live  or  historical  environmental  data. 


Phase:  SRR  SFR  PDR  CDR 


Design/ 

Payload 

Maturity: 

(w/Models) 


High  level  need: 
Aircraft 


Mid  level  need: 
take  off,  land,  fly 


v&v  Operational 

Focus*  level  moc*e's 


High  level 

performance.  (Aero, 
some  P&FQ) 


Lower  level  need:  Lowest  level  need: 

Employ  legacy  weapons  employ  advanced 

weapons;  stealth,  etc. 


Macro-level  integration,  Full  integration 

some  system  and  systems 

functionality,  full  P&FQ  functionality 


Surrogates,  traditional 
materials,  hardware,  processes 


Base  airframe  with  some  advanced  materials  Final  Config:  advanced  materials 

(composites)  hardware  (SIL  assets)  (composites/exotics)  advanced 

hardware,  final  avionics 


•‘Derived  from  Ernest  S.  "Turk"  Tavares,  Jr.  and  Larry  Smith 


Figure  3.  Dynamic  Models  and  Surrogates  to  Support  Continuous  "Virtual  V&V"  Early  in  the  Lifecycle 


Increasingly  there  will  be  much  more  complexity  in  the  software  prior  to  PDR  and  CDR,  and  this  creates 
different  concerns  for  NAVAIR  from  prior  generations  of  air  vehicle  systems.  Testing  will  be  required  to 
ensure  that  the  continuously  refined  representation  of  the  system  models  and  implementation  meet  the 
timing  (temporal  interactions)  and  performance  needs.  This  will  including  testing  not  only  the  system 
itself,  buttesting  required  to  verify  the  models  and  simulation  are  integrated  and  working  with  the  system 
as  it  is  refined  over  time. 


Mission-Level  Simulation  Integration  with  System  Simulation  and  digital  Assets 

The  previous  two  examples  reflect  on  MCE  as  it  is  applied  at  the  system  level,  which  was  the  scope  for 
our  research.  However,  several  organizations  discussed  mission-level  simulation  capabilities,  including 
NAVAIR.  One  organization  demonstrated  mission-level  simulations  that  are  being  integrated  with  system 
simulation,  digital  assets  and  aircraft  products  providing  new  types  of  web-based  services.  The 
organization  provided  a  live  (with  some  artificial  data)  multi-scenario  SoS  demonstration  that  runs  on  a 
modeling  and  simulation  (M&S)  infrastructure  that  integrates  with  other  M&S  capabilities  as  well  as  live 
products  that  can  be  hosted  within  a  cockpit  or  operate  through  server  and  web-based  services.  The 
scenarios  represented  a  commercial  version  for  a  DoD-equivalent  mission  analyses.  The  M&S 
infrastructure  is  used  to  both  analyze  new  types  of  services  that  can  be  added  to  their  portfolio  and  can 
be  used  to  demonstrate  to  potential  customers  capabilities  using  real  or  artificial  data.  These  capabilities 
are  used  in  a  way  that  improves  their  own  systems  and  capabilities,  but  they  use  these  capabilities  to 
solicit  inputs  from  potential  customers  on  new  types  of  products  and  services.  However,  even  with  the 
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advancements  this  organization  discussed,  there  are  some  challenges  with  developing  the  integrations 
as  there  was  not  a  grand  architectural  scheme  or  plan  when  they  first  started  developing  the  underlying 
infrastructure.  We  emphasize  this  point,  because  organizations  need  to  apply  system  engineering 
principles  to  their  MCE  engineering  environment. 


3D  Environments  and  Visualization 

Several  organizations  demonstrated  (or  showed  results  from)  some  of  their  3D  and  visualization 
capabilities  [49],  One  organization  discussed  and  demonstrated  the  use  of  two  different  types  of  3D 
environments  for  both  customer  engagements  and  for  on-going  (often  daily)  design  engineering  analysis 
and  review  sessions  in  3D  environments.  The  organization  does  use  commercial  technologies  but  have 
developed  a  significant  amount  of  infrastructure  on  their  own.  Several  similar  stories  were  recorded  from 
others  about  the  need  to  develop  unique  infrastructure.  A  visit  was  made  to  the  Cave  Automated  Virtual 
Environment  (CAVE),  as  shown  in  Figure  4,  where  we  were  immersed  in  a  virtual  3D  environment  that  is 
used  for  both  analysis  and  design  review  purposes. 


Dynamic  Operational  Views  for  Mission  and  System  Simulation  and  Analysis 

Extending  on  the  example  shown  in  Figure  2,  there  are  modeling  environments  and  frameworks  to  create 
dynamic  Operational  Views  (e.g.,  an  OV1),  which  help  to  understand  and  characterize  the  Mission  Context 
for  the  needed  System  Capabilities,  as  shown  in  Figure  5.  In  traditional  DoDAF  models,  static  OVls  are 
used,  but  the  newer  capabilities  provide  for  dynamic  operational  scenarios  that  not  only  allow  for 
analysis,  but  also  are  being  leveraged  as  scenarios  for  testing.  In  many  instances  these  types  of  capabilities 
have  integrations  with  other  types  of  models,  simulation  and  analysis  capabilities.  The  current  state 
shows  that  MCE  is  moving  beyond  static  DoDAF  views.  The  computational  and  visualization  technologies 
bring  the  behavioral  views  into  perspective,  but  can  increasingly  bring  the  temporal  aspects  into  play. 
This  is  a  capability  that  can  be  used  at  the  mission  level  as  well  as  the  system  level.  Continuing  on  the 


3  Image  credit:  image  credit:  media.gm.com 
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theme  of  integration,  these  capabilities  are  packaged  in  such  a  way  to  integrate  with  other  system 
engineering  and  multi-physics  simulation  capabilities  [115]. 


Environment  Models  Comms  &  Radar  Models 

Model  terrain,  atmosphere  A  space  Model  RF  propagation  A  interference 


Figure  5.  Dynamic  OV1  with  Integrations  to  Other  Models  and  Digital  Assets4 


Multidisciplinary  Design,  Analysis  and  Optimization  (aka  Tradespace  Analysis) 

One  organization  gave  a  demonstration  in  a  Multidisciplinary  Design,  Analysis  and  Optimization  (MDAO) 
Laboratory.  This  organization  (call  them  ABC)  mentioned  that  several  years  back  they  had  a  consulting 
organization  assess  their  state  of  the  practices  against  other  Industry  contractors  and  it  was  believed  that 
ABC  was  trailing  the  others  in  the  use  of  MDAO  capabilities.  They  decided  that  they  did  not  need  to  do  a 
Return  on  Investment  analysis  and  just  moved  forward  with  putting  their  lab  together.  The  information 
they  presented  showed  that  they  have  a  much  more  comprehensive  approach  today;  this  includes  both 
integration  of  the  tools,  and  the  methodological  approaches.  They  established  the  information 
technology  infrastructure  to  facilitate  the  integrations  across  the  design  space  of  many  facets 
Aerodynamics,  Mass  Properties,  Performance,  Propulsion,  Operational  Analysis,  Ops-support, 
Manufacturing,  and  assembly  and  lifecycle  costs  across  multi-mission  scenarios,  but  not  necessarily  cross¬ 
domain  at  the  same  time.  They  are  systematic  about  creation  of  design  of  experiments.  They  stated  that 
in  the  use  of  these  technologies,  they  often  uncover  or  expose  things  that  are  not  intuitive  -  that  is  the 
more  comprehensive  analysis  allows  for  many  more  excursions  and  they  can  uncover  issues  early.  The 
power  from  the  automation  and  efficiency  of  the  tools  often  allows  them  to  spend  more  time  doing  more 
in-depth  analysis;  they  stated  that  they  often  do  100  times  as  many  excursion  of  the  design  space  with 
MDAO  versus  traditional  manual  methods. 


4  Image  credit:  AGI 
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Figure  6  provides  a  comparison  of  a  Legacy  approach  to  Multidisciplinary  Design  Optimization  (MDO)  [51], 
analogous  to  Multidisciplinary  Design  Analysis  and  Optimization  (MDAO).  The  comparison  is  based  on, 
and  shows: 

■  Setup  time  is  longer  (specification)  for  MDO 

■  Execution  time  is  shorter  for  MDO 

■  Management  time  is  significantly  shorter  for  MDO 

■  Reasoning  time  is  longer  for  MDO 

■  Number  of  possible  iterations  is  orders  of  magnitude  more  in  the  same  amount  of  total  time 


(e.g.  determining  tasks,  (e.g.  generating  options  (e.g.  representing,  documenting  (e  g.  interpreting  results, 

staffing,  and  what  information  and  running  analyses)  and  coordinating  existing  choosing  options) 

is  used  and  produced)  information 


Figure  6.  MDAO  Compared  with  Legacy  Tradespace  Analysis  [51] 

Once  the  experimentation  framework  is  setup  [51],  the  case  study  group  was  able  to  do  500  times  as 
many  iterations  to  examine  various  different  alternatives  in  the  same  amount  of  time  as  with  the  legacy 
(manual  approach).  We  heard  similar  stories  during  our  organizational  discussions  that  were  using  MDAO. 
This  supports  data-to-decisions  with  engineering  technical  data  and  information  that  is  produced,  not 
just  documents.  There  are  both  commercial  and  open  source  MDAO  tools  and  environments  that 
integrate  with  different  modeling  tools  and  many  solvers. 


Interactive  Successful  Refinement  of  Design  Space 

The  DARPA  META  Advance  Vehicle  Make  (AVM)  program  also  used  MDAO  for  continuous  and  iterative 
successive  refinement  of  tradespace  alternatives,  with  considerations  for  manufacturability  leading  to 
"executable  requirements"  with  continuous  test  at  increasing  levels  of  fidelity  [7],  AVM  Philosophical 
Underpinnings  document,  are  quoted  verbatim  below,  the  objectives  were: 

"Raise  the  level  of  abstraction  such  that  the  designer  need  not  manipulate  the  design  at  the  lowest 
numbered  part  level,  but  can  operate  at  varying  levels  of  hierarchical  abstraction  and  model  fidelity;" 

"Develop  practical  and  observable  metrics  of  complexity  to  augment  size,  weight,  power,  and 
performance  in  informing  design  decisions;" 

"Enable  rapid  exploration  of  the  design  trade-space  for  high-fideiity  requirements  trade-offs;" 
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"Yield  detailed  system  designs  that  are  "correct-by-construction,"  i.e.,  probabilistically  verified  for 
consistency,  multi-mode  interactions,  and  first-order  performance  characteristics  across  all  the 
relevant  physics  domains  (including  embedded  software)." 

As  shown  in  Figure  7,  the  overall  approach  provides  evidence  for  the  technical  feasibility  argument,  and 
another  example  that  relates  to  our  proposed  vision  concept,  namely: 

■  A  more  continuous  and  iterative  approach  using  successive  refinement  of  tradespace 
alternatives  spanning  many  model  types  across  multiple  disciplines 

■  With  data  visualization 

■  With  considerations  for  manufacturability 

■  Leading  to  "executable  requirements" 

■  With  continuous  test  at  increasing  levels  of  fidelity 
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Figure  7.  DARPA  META  Concept 

There  were  examples  of  extreme  acceleration  through  HPC  where  the  software  allows  for  doing  different 
types  (four-levels)  of  blast  analysis  (for  tanks);  reduced  time  from  months  to  3-days  of  computing  time, 
and  a  few  lessons  learned. 


ID,  2D,  &  3D  Model  Creation,  Simulation,  Analysis  for  Physics-based  Design 

Discussion  with  many  organizations  on  ID,  2D  and  3D  model  creation,  simulation,  analysis,  and 
management  capabilities  focused  primarily  on  physics-based  design  with  increasing  support  for  cross¬ 
domain  analysis.  Some  organizations  have  unique  capabilities  and  there  is  an  increasing  trend  to  support 
broader  cross-domain  analysis  through  better  integration  of  different  types  of  models.  Some  models 
allow  for  the  plug-in  of  niche-capability  libraries  and  solvers,  using  a  platform-based  approach  to  create 
more  of  an  ecosystem  (i.e.,  "think  apps").  While  yet  other  models  are  customizable  to  leverage  High 
Performance  Computing  (HPC);  that  is,  they  have  been  programmed  to  take  advantage  of  parallel 
computation.  There  are  challenges,  but  also  advances  in  model  transformation  and/or  interoperability, 
and  the  need  for  formalized  semantics  is  known.  There  are  also  multiple  suppliers  that  often  provide  a 
suite  of  tools  that  cover  different  parts  of  the  lifecycle,  but  are  not  necessarily  integrated. 
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Modeling  and  Simulation  Integration  with  Embedded  System  Code 


There  were  many  relevant  topics  that  support  the  vision  of  model-centric  engineering,  including  one 
discussion  by  an  organization  that  performs  modeling  and  simulation  of  the  flying  qualities  that  integrate 
directly  with  the  code  generated  from  the  Simulink  model  for  the  control  laws  of  an  actual  aircraft.  By 
using  the  actual  control  law  software  that  flies  the  aircraft  with  the  modeling  and  simulation,  this 
increases  the  accuracy  and  confidence  in  the  results.  This  also  acts  as  a  type  of  validation  on  the  control 
laws  software  against  flying  quality  simulation  scenarios.  Finally,  the  combined  integration  of  the 
modeling  and  simulation  with  the  actual  code  adds  to  the  arguments  about  the  integrity  and  trust  in  the 
models. 


Platform-based  Approaches 

Platform-based  approaches  are  used  by  commercial  tool  vendors  as  well  as  developer  of  systems.  The 
system  developers  use  virtual  integration  to  help  refresh  systems  and  do  system  upgrades  on  periodic 
schedules,  which  in  many  cases  is  business  critical  to  companies.  For  example,  two  organizations  in  the 
automotive  space  discussed  platform-based  approaches  that  are  tactically  driven  by  the  periodic  cycles 
demanded  for  sales  roll  outs  at  12,  18,  24,  30,  and  36  month  delivery  cycles.  In  the  12  and  18  months 
cycle  they  might  change  feature  colors  but  every  interface  is  exactly  the  same  with  no  electric  changes. 
At  24  months  they  may  make  some  minor  changes  including  electrical.  At  30  months  changes  to  the  types 
of  subsystem,  components,  for  example,  Figure  8  is  based  on  approach  that  uses  the  standards  Modelica 
and  Functional  Mockup  Interface  (FMI),  which  support  co-simulation  and  cross-domain  analysis.  Finally 
at  36  months  a  complete  redesign  usually  occurs. 


•ngin« 


Modelica  vehicle  system  model 


world  #  road 


Figure  8.  Vehicle  System  Platform-based  Model 


There  are  several  implications  from  these  scenarios,  and  we  will  list  two: 

■  The  automotive  industry  has  been  effective  at  standards-based  approaches  to  platform  based 
development  and  deliveries,  which  are  critical  to  cycle  times  and  their  business 
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o  They  also  use  their  own  customized  and  non-standards  based  approaches  to  support 
platform-based  development 

■  The  standards-based  approach  for  integrating  models  across  various  domain  using  the 

Functional  Mockup  Interface  (FMI),  provides  a  capability  as  well  as  a  metaphor  that  would  be  a 
way  for  government  to  collaborate  with  contractors  across  the  domains  of  a  system  by  using  an 
FMI-like  backplane  to  support  dynamic  analysis  and  co-simulation  of  one  or  more  integrated 
capabilities 

o  Functional  Mockup  Units  can  be  provided  in  either  model-based  form,  or  can  be  provided  in 
binary  form;  the  binary  form  would  allow  contractors  to  share  their  dynamic  capabilities, 
but  also  protect  their  intellectual  property 


Modeling  and  Simulation  Reducing  Physical  Crash  Testing 

The  automotive  companies  stated  that  modeling  and  simulation  is  being  used  to  significantly  reduce  crash 
testing.  Some  mentioned  reducing  cash  testing  from  320  down  to  80  crash  tests.  This  is  of  particular 
interest  since  destructive  testing  and  other  types  of  testing  can  be  expensive  and  time  consuming.  The 
key  implication  for  our  sponsor  is  that  simulation  may  be  able  to  reduce  or  at  least  be  targeted  to  specific 
types  of  flight  tests  where  there  is  the  greatest  uncertainty  about  performance  or  margins. 


Modeling  and  Simulation  of  the  Manufacturing 

The  DARPA  META  project  [7]  and  other  industry  organizations  discussed  factoring  in  manufacturability 
constraints  during  the  MDAO  of  the  design  space.  Some  simulate  the  manufacturing  processes  in  advance 
of  developing  the  tooling.  One  organization  discussed  model-based  manufacturing,  model-based 
inspection,  design  for  manufacturability,  additive  manufacturing,  their  smart  manufacturing  efforts,  and 
advanced  design  tooling  (modeling  and  simulation  infrastructure).  In  addition,  the  set-based  design 
concept  originally  attributed  to  Toyota  described  how  the  design  and  manufacturing  processes  work 
more  concurrently.  These  concepts  are  strongly  related  to  tradespace  analysis  and  design  optimization. 
This  may  also  provide  a  means  for  reducing  the  time  to  develop  large  air  vehicle  systems. 

A  rule  of  thumb  is  that  the  effort/time  to  get  from  Milestone  A  to  Critical  Design  Review  (CDR)  is  about 
30  percent  of  the  total  time,  where  the  time  from  CDR  to  Initial  Operating  Capability  (IOC)  is  about  70 
percent  of  the  total  time.  With  some  of  the  new  approaches  to  produce  digital  information,  which 
considers  modeling  and  simulation  analysis  of  manufacturability  prior  to  CDR,  the  digital  information  at 
CDR  could  significantly  reduce  the  70  percent  effort  from  CDR  to  IOC,  which  also  builds  to  the  argument 
for  being  able  to  reduce  the  acquisition  time  by  25  percent  with  MCE. 


Workflow  Automation  to  Subsume  Process 

Automated  workflows  arose  from  the  manufacturing  world.  A  key  idea  is  that  if  we  could  formalize  all 
the  model-centric  artifacts,  including  the  process,  we  could  "generate"  information  to  drive  a  workflow 
engine  that  could  significantly  subsume  the  process.  This  would  enable  NAVAIR  to  make  decisions  in  real¬ 
time  that  are  completely  data-driven.  This  would  also  make  decision-making  more  standardized  and 
reduce  the  dependence  of  decision  quality  based  on  individual  experience.  The  process  would  be 
autonomous  and  adaptive,  and  coordination  would  replace  planning. 

■  Workflow  automation  has  the  potential  to  subsume  the  entire  process;  everything  driven  by 
data,  data  dependencies;  this  would  be  towards  a  "radical  transformation" 
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o  The  key  reason  for  this  concern/question  is  that  the  effort  in  modeling  the  "As  Is"  process  is 
reflecting  that  it  is  potentially  too  difficult  to  ever  fully  create  or  follow  a  document-driven 
process 

■  To  a  lesser  degree,  there  are  other  types  of  products  that  provide  workflow  automation  support 
integrations  for  work  such  as  design  optimization 

o  We  spoke  with  both  commercial  companies  that  provide  these  capabilities  and  industry 
companies  that  use  these  technologies 

o  They  do  help  speed  up  and  make  the  design  optimizations  more  efficient,  and  allow  for 
more  iterations,  and  more  systematic  regression  analysis 

A  DARPA  META  project  ARRoW  (Adaptive,  Reflexive,  Robust  Workflow)  developed  their  concept  for  a 
single  source  of  technical  truth  [6],  This  capability  allowed  them  to: 

■  Automatically  generate  workflows  based  on  the  data/information  dependencies 

■  Integrate  model  measures  that  are  "automatically"  and  continuously  generated  -  this  relates  to 
a  new  concept  of  model  measures 

■  Incorporate  knowledge  through  the  use  of  ontologies  and  patterns  as  part  of  the  model 
reference  (reference  architecture) 


Single  Source  of  Truth,  Modeling  Patterns,  Ontologies,  Profiles,  Continuous  Integration 

We  heard  several  organizations  discuss  the  concept  of  a  single  source  of  truth5.  For  example,  NASA/JPL 
provided  a  perspective  on  their  concept  and  evolving  instantiation  of  a  "Vision"  model  [8],  They  have 
modeled  and  are  formalizing  the  overarching  Model-based  Engineering  Environment  (MBEE)  [43] 
(designing  system)  being  used  to  develop  instances  of  a  system  as  well  as  the  mission  characterization 
that  is  captured  in  a  system  model.  They  continue  to  formalize  the  modeling  methodology  through  model 
patterns  [33]  [79]  that  are  captured  through  ontologies,  which  are  associated  with  a  tool-based  approach 
that  not  only  guides  development,  but  provides  model  analysis  to  ensure  compliance  with  the  patterns 
(e.g.,  models  are  well-formed,  consistent,  etc.)  [69],  This  provides  their  foundation  for  a  single  source  of 
truth  that  is  used  both  for  development  and  continuous  reviews. 

Among  other  topics  mentioned  previously,  NASA/JPL  has  developed  an  Architecture  Framework  Tool 
(AFT)  for  Architecture  Description  [9],  which  provides  an  overarching  perspective  on  one  of  the  views 
needed  for  our  Task  3  vision  model,  and  is  partially  supported  with  their  evolving  Open-MBEE  [43], 

These  two  concepts  are  further  supported  with  a  rigorous  approach  to  systems  engineering  (SE); 
NASA/JPL  have  identified  about  25  modeling  patterns  applicable  to  systems  engineering.  They  formalize 
the  patterns  in  ontologies  using  Web  Ontology  Language  (OWL)  [125]  to  provide  a  way  of  defining  a  set 
of  concepts  and  properties  applicable  to  the  domain  of  discourse;  in  this  case  not  about  the  space  domain, 
but  about  the  SE  domains  for  concepts  such  as:  component,  function,  requirement,  and  work  package, 
data  properties  like  mass  and  cost,  and  object  properties  (relationships)  like  performs,  specifies,  and 
supplies.  This  provides  for  a  controlled  vocabulary  and  enforcing  rules  for  well-formedness,  which 
permits,  among  other  things,  interdisciplinary  information  integration,  and  automated  analysis  and 
product  generation.  Because  the  SE  ontologies  are  expressed  in  OWL,  they  are  amenable  to  formal 
validation  (syntactic  and  semantic)  with  formal  reasoning  tools.  The  approach  embedded  in  SysML  and 
the  OWL  ontologies  is  created  by  transformations  from  SysML  models  [69],  Once  a  model  is  completed 
other  transformations  are  performed  to  the  model,  such  as  checking  properties  of  well-formedness  and 


5  NAVAIR  uses  Single  Source  of  Technical  Truth. 
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consistency  of  the  model.  They  currently  have  about  60,000  test  cases  for  checking  these  types  of 
properties.  The  approach  is  illustrated  in  several  case  studies  [79], 


Quantification  of  Margins  and  Uncertainty 

Sandia  National  Laboratory  (Sandia)  discussed  some  of  the  most  advanced  approaches  for  supporting 
uncertainty  quantification  (UQ)  to  enable  risk-informed  decision-making.  The  information  they  provide 
reflects  on  the  advanced  nature  of  their  efforts  and  continuous  evolution  through  modeling  and 
simulations  capabilities  that  operate  on  some  of  the  most  powerful  high  performance  computing  (HPC) 
resources  in  the  world.  We  heard  about  their  HPC  capabilities,  Common  Engineering  Environment, 
methodologies  on  Quantification  of  Margins  and  Uncertainty  (QMU)  [87]  and  an  enabling  framework 
called  Dakota  [109],  which  should  contribute  to  our  Risk  Framework  (Task  4).  Sandia's  team  also  discussed 
the  various  modeling  and  simulation  capabilities  and  resources  that  run  on  the  HPC.  This  ultimately  led 
into  a  discussion  about  Model  Validation  and  Simulation  Qualification  [106], 


Gaps  and  Challenges 

During  our  site  visits,  we  asked  the  organizations  to  share  some  of  the  gaps  and  critical  challenges  too, 
and  several  of  them  we  have  been  highlighting  in  the  previous  summary.  Some  provided  inputs  beyond 
the  question  of  "technical  feasibility,"  for  example: 

■  Mission  complexity  is  growing  faster  than  our  ability  to  manage  it 

■  Not  identifying  design  or  integration  problems  until  late  in  lifecycle 

o  Complex  systems  have  greater  cross-domain  dependencies,  and  many  of  the  modeling  and 
simulation  efforts  are  not  doing  analysis  in  terms  of  the  integration  of  models  and  their 
associated  simulations 

■  Growth  and  complexity  of  software  and  the  verification  challenges,  which  are  essential  to 
airworthiness  and  safety 

■  Inability  to  share  models  in  a  collaborative  environment 

o  This  point  again  may  relate  to  the  underlying  semantics  of  models  in  specific  domains  that 
are  not  easily  shared 

■  Use  of  unvalidated  models  in  simulations  leading  to  incorrect/invalid  results 
o  How  do  we  validate  models,  especially  if  there  is  an  explosion  of  models 
o  This  is  part  of  the  model  "integrity"  need 


Cross-Domain  Interoperability 

MCE  technologies  enable  more  automation  and  efficiencies;  however,  there  are  only  a  few  approaches 
that  deal  with  cross-domain  model  interoperability  and  consistency,  and  the  appropriate  use  of 
methodologies  is  necessary  to  make  these  technologies  effective.  A  DARPA  META  research  project  cited 
related  challenges,  such  as  the  heterogeneity  and  diversity  of  domain  engineering  tools,  and  the  span  of 
design  flow  activities  essential  for  design  of  complex  CPS;  they  further  noted  that  the  industrial  state  of 
the  art  consists  of  islands  of  model-based  automation,  but  supported  by  tool  verticals  where  integration 
is  limited  [7],  One  DARPA  META  project  team  developed  openMETA6,  an  architecture,  and  CyPhy  a  new 
model  integrating  language  to  investigate  the  potential  to  integrate  models  and  tools  in  a  semantically 
precise  way.  Their  report  suggests  successes  but  also  several  other  challenges.  For  example  a 


6  The  authors  provide  examples,  but  do  not  promote  any  particular  approach  or  tooling. 
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fundamental  aspect  of  their  approach  involved  iterative  and  successive  refinement  of  a  design  space 
through  tradespace  analysis  of  assembled  component  libraries.  It  was  determined  that  when  component 
libraries  were  not  developed  with  an  appropriate  methodological  approach  that  aligned  with  the 
capability  of  the  openMETA  tools,  the  approach  and  system  was  not  as  effective  as  planned.  This  led  to 
effort  to  re-model  the  component  libraries  in  order  to  demonstrate  the  new  approach  and  tools.  In  such 
a  system  any  set  of  components  often  needs  to  satisfy  multiple  constraints  as  it  relates  to  other 
components  ranging  from  relatively  simple  structural  constraints  to  fluid  dynamics,  thermal,  software 
control,  reliability,  safety,  cost,  and  others. 

The  openMETA  research  demonstrated  the  potential  for  automated  selection  of  components.  However, 
the  lessons  learned  suggests  that  the  specification  of  component  properties  and  operational  constraints 
must  be  aligned  with  the  disparate  analysis  capabilities  of  one  or  more  modeling  and  analysis  tools  that 
were  not  designed  to  be  integrated  [7],  Therefore  tools  alone,  even  if  integrated,  are  not  enough;  the 
appropriate  modeling  methodologies  are  also  important  to  leverage  the  analytical  and  generative 
capabilities  of  MCE  tools. 


Complexity  of  Software  and  the  Verification  Challenge 

The  strict  requirement  for  safety  and  airworthiness  for  the  NAVAIR  air  vehicle  systems  requires 
comprehensive  rigor  in  verification.  As  90  percent  of  the  functionality  of  in  a  5th  generation  air  vehicle 
system  is  in  software  that  implies  a  significant  amount  of  software  verification. 

One  particular  challenge  that  we  discuss  in  the  meetings  with  organizations  is  software.  Jaime  Guerrero, 
one  of  our  NAVAIR  sponsors  that  attended  every  organizational  meeting,  usually  discusses  his  effort  on 
the  Joint  Strike  Fighter  (JSF)  stating  that:  "90  percent  of  the  functionality  in  5th  generation  air  vehicle 
systems  (e.g.,  F-35)  is  in  software."  There  are  reports  that  software  testing  is  taking  a  long  time  (GAO 
report  [53]).  While  there  is  use  of  models,  the  detailed  software  behavior  is  often  written  manually,  which 
minimizes  the  ability  to  formalize  analysis,  generate  the  code,  and  automate  test,  with  the  possible 
exception  of  Simulink  (but  not  everything  is  modeled  like  a  control  system).  This  is  one  of  the  greatest 
concerns  to  the  goal  of  reducing  25  percent  of  the  time. 

To  put  this  challenge  into  perspective,  NASA  presented  industry  data  indicating  that  verification  is  88 
percent  of  the  cost  to  produce  DO-178B  Level  A7  software,  and  75  percent  for  Level  B  software  [29],  These 
types  of  verification  requirements  are  required  for  many  aspects  of  NAVAIR  vehicles,  such  as  the  control 
laws  for  the  F-35.  As  shown  in  Figure  9,  the  DARPA  META  pre-program  solicitation  (META)  describes  how 
continually  increasing  complexity  impacts  the  verification  costs  of  software  and  delivery  time  [15].  META 
claims  that  the  fundamental  design,  integration,  and  testing  approaches  have  not  changed  since  the 
1960s,  as  shown  in  Figure  10.  The  META  program  goal  is  to  significantly  reduce,  by  approximately  a  factor 
of  five,  the  design,  integration,  manufacturing,  and  verification  level  of  effort  and  time  for  cyber  physical 
systems.  The  complexity  has  increased  for  integrated  circuits,  as  it  has  for  software-intensive  systems, 
but  the  developers  of  integrated  circuits  have  maintained  a  consistent  level  of  effort  for  the  design, 
integration  and  testing  efforts,  as  reflected  in  Figure  9.  The  need  is  to  understand  key  reasons  why 
software-intensive  systems  production  is  different  from  integrated  circuits.  One  fundamental  difference 
is  that  software  behavior  requires  nonlinear  operations  and  constraints  that  are  implemented  on 
computing  hardware  where  operations  are  performed  and  results  stored  in  floating  point 
representations.  This  makes  the  automated  verification  problem  more  challenging  than  for  integrated 


7  DO-178B/C  is  the  Software  Considerations  in  Airborne  Systems  and  Equipment  Certification  document  dealing 
with  the  safety  of  software  used  in  certain  airborne  systems.  Level  A  is  a  characterization  for  the  most  safety- 
critical  aspects  of  the  software,  and  required  a  more  comprehensive  amount  of  software  verification. 
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circuits,  where  automated  verification  and  analysis  is  based  primarily  on  logic  or  bit-level  manipulations. 
Chip  developers  used  to  rely  on  simulation,  much  like  software  development  uses  debugging  and  manual 
testing,  but  the  chip  verification  would  cost  more  than  50  percent  of  the  effort  and  defects  that  escape 
to  the  field  could  cost  $500M8.  They  now  rely  more  on  formal  methods  and  tools  to  support  development 
and  verification. 


Complexity’ 

[Part  Count  +  Source  Lines  of  Code  (SLOC)] 

Note  (*):  Not  a  great  metric.  But  that’s  what  we  have  today.  META  will  come  up  with  better  metrics. 


2 


Figure  9.  DARPA  META  Program9 


8  http://en.wikipedia.org/wiki/Pentium  FDIV  bug 

9  DARPA  META  program  APPROVED  FOR  PUBLIC  RELEASE.  DISTRIBUTION  UNLIMITED 
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MIL-STD-499A  (1969)  Systems  Engineering  Process -As  Employed  Today 
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Figure  10.  META  Program  Claims  Conventional  V&V  Techniques  do  not  Scale  to  Highly  Complex  Systems10 


There  may  be  many  differences  between  hardware  and  software,  and  we  briefly  summarize  the  points: 

■  Software  behavior  often  relies  on  floating  point  variables  with  nonlinear  relationships  and 
constraints,  which  is  not  the  case  in  integrated  circuits 

■  This  requires  different  mechanisms  for  analysis  and  verification  than  are  used  in  hardware 
analysis  and  verification 

■  Other  than  models  like  Simulink,  the  detailed  software  behaviors  (functions)  are  still  written 
mostly  by  hand,  limiting  automated  analyses 

o  Some  discuss  the  use  of  automated  generation  of  code 

o  But  many  are  using  coding  frameworks,  which  can  generate  the  code  structure,  but  the 
detailed  behavior  is  written  in  the  code  using  languages  like  C++ 
o  Newer  approaches  that  rely  on  domain-specific  modeling  are  being  researched  through 
DARPA  efforts,  but  most  have  not  become  mainstream  [22],  [99], 

Finally,  we  had  discussions  with  organizations  that  are  researching  the  use  of  quantum  computing 
focused  on  addressing  the  ever-increasing  challenge  of  verification  and  validation  (V&V)  in  systems  that 
are  increasing  in  complexity.  They  stated  that  V&V  costs  are  growing  at  the  fastest  rates  of  any  system 
component  and  rates  are  expected  to  accelerate  with  exponential  growth  in  software  size  and  system 
complexity  driving  exponential  growth  in  certification  costs.  These  types  of  technological  breakthroughs 
can  also  be  factored  into  our  scenarios  that  model-centric  engineering  will  change  how  we  work,  and  that 
will  reduce  or  eliminate  some  challenges. 


Metrics  and  Tools  for  Verification  and  Validation  of  Cyber-Physical  System 

The  National  Institute  of  Standards  and  Technology  (NIST)  Foundations  for  Innovation  in  Cyber-Physical 
Systems  report  [84]  as  well  as  the  European  ARTEMIS  Research  agenda  [4]  points  out  similar  needs,  as 


10  DARPA  META  program  APPROVED  FOR  PUBLIC  RELEASE.  DISTRIBUTION  UNLIMITED 
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stated  in  the  previous  paragraph,  across  many  CPS  domains.  The  NIST  report  identifies  21  barriers  and 
challenges  for  CPS  reliability,  safety,  and  security.  In  the  top  rated  category  of  Metrics  and  Tools  for  CPS 
Verification  and  Validation  (V&V)  the  NIST  report  cites  the  top  three  challenges  as:  1)  the  need  for 
increasing  coverage  of  V&V  while  reducing  costs,  2)  coping  with  complexity  and  scale  of  systems  when 
performing  V&V,  and  3)  the  inability  to  apply  formal  methods  at  appropriate  abstraction  levels.  The 
hidden  challenge  again  for  CPS  is  that  V&V  must  be  comprehensive  across  the  related  domains. 

History  suggests  that  there  is  limited  success  in  exposing  cross-domain  issues  until  a  physical  integration 
is  developed  to  enabling  integration  and  system  testing.  In  the  case  of  systems  such  as  the  Joint  Strike 
Fighter  (JSF)  this  has  resulted  in  program  delays.  This  could  be  due  to  complexities,  leading  to  late 
discovery  during  integration  and  test  related  to  need  capabilities  implemented  across  the  domains  that 
cannot  be  captured  using  document-centric  requirements.  There  is  still  research  needed,  because  there 
is  not  one  clear  choice  for  a  method  and  toolset  to  support  model  integration  across  the  domains  of  a 
system  like  JSF,  which  is  needed  for  design  and  early  virtual  V&V  of  a  CPS.  This  is  especially  important  to 
acquisition  organizations  that  perform  the  role  of  Lead  System  Integrator. 


Summary 

We  wanted  to  provide  some  evidence  of  traceability  to  different  instances  of  use  of  MCE- re  levant  and 
cross-domain  technologies.  We  highlight  evidence  (marked  as  an  X)  if  we  actually  had  discussions  or 
demonstrations  related  to  this  technology  as  shown  in  Figure  11.  Due  to  sizing  constraints,  this  matrix 
includes  only  a  subset  of  the  organizations  that  were  involved  in  Task  1.  We  also  relate  these  to  their 
possible  impacts/relationships  on  characteristics: 

■  Performance 

■  Integrity 

■  Affordability 

■  Risk 

■  Methodology 

■  Single  Source  of  Technical  Truth 

Lastly,  we  relate  them  to  engineering  efforts  discussed  in  the  kickoff  meeting  for  this  research: 

■  Prioritization  &  Tradeoff  Analysis 

■  Concept  Engineering 

■  Architecture  &  Design  Analysis 

■  Design  &  Test  Reuse  &  Synthesis 

■  Active  System  Characterization 

■  Human-System  Integration 
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Figure  11.  Traceability  and  Scope  of  Data  Collection  for  Task  1 


When  all  of  the  relevant  technical  factors  are  considered  that  are  critical  to  a  SE  radical  transformation 
through  MCE  or  more  generally  digital  engineering,  the  technical  feasibility  question  seems  to  place 
emphasis  on  three  critical  items: 

1.  Cross-domain  and  multi-physics  model  integration,  and  the  associated  methodologies 

2.  Technologies  to  establish  and  quantify  model  integrity 

3.  High  Performance  Computing  (HPC),  which  enables  1  and  2 

Potentially,  even  more  fundamental  is  the  need  for  a  single  source  of  technical  truth  (SSTT).  The  SSTT  is 
an  enabler  for  cross-domain  interoperability  needed  for  multidisciplinary  design,  analysis  and 
optimization  for  problem  and  design  space  exploration.  The  SSTT  requires  that  all  information  used  to 
assess  performance  is  semantically  consistent  with  MCE  technologies  and  methods  used  for  assuring 
integrity  and  the  orchestrated  workflow  is  data-driven  (not  process  driven). 

Model  integrity  is  defined  as  verifying  our  understanding  of  the  models'  margins  and  uncertainty  in  what 
they  "predict"  or  in  other  words  when/how  do  we  trust  the  models.  The  ability  to  use  logged  data  to 
continually  calibrate  models/simulation  is  desired.  Cross-domain  integration  of  models  may  also  be  a  way 
to  have  greater  confidence  in  simulation  models  because  results  of  the  models  should  be  closer  to  the 
real  system  data. 

As  another  example,  the  Engineered  Resilient  System  (ERS)  research  effort  builds  on  some  of  the  themes 
discussed,  showing  how  such  an  approach  and  support  for  data-driven  decision  can  subsume  process,  as 
shown  in  Figure  12  [63]: 

■  Single  Source  of  Technical  Truth  (SSTT) 

■  Appropriate  views  and  viewpoints  for  the  different  stakeholders 

■  Multidisciplinary  Design,  Analysis  and  Optimization  (MDAO) 

■  Workflow  Orchestration  -  by  having  the  data  dependencies  being  known  within  the  SSTT 

■  Enabled  by  High  Performance  Computing 
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Figure  12.  Engineering  Resilient  System  Conceptual  Representation  of  Environment  [63]  -  Enhanced11 


11  This  figure  has  been  enhanced  to  highlight  the  notions  of:  1)  Appropriate  Views  for  Stakeholders,  2)  Single 
Source  of  Technical  Truth,  3)  Multidisciplinary  Design,  Analysis  and  Optimization,  and  4)  Enabled  by  High 
Performance  Computing 
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Part  II:  Previous  Work  Material 


The  material  in  the  remainder  of  the  document  has  been  extended  or  refined  from  the  RT-48/118  Final 
Technical  Reports  [21]  [22],  Additional  details  related  to  this  material  have  been  documented  in  the  bi¬ 
monthly  status  or  working  session  meeting  minutes. 


Task  1  -  Assessing  the  State-of-the-Art  MBSE 


This  section  summarizes  some  information  about  the  visits  and  discussions  with  industry,  government, 
and  academia.  Prior  to  each  meeting,  we  sent  our  coordinator  package  to  an  organization  coordinator; 
the  package  explains  the  overarching  goals  of  the  research  task.  We  often  iterate  with  the  organization 
about  the  agenda.  In  coordinating  the  agenda  with  our  organizing  hosts  and  at  the  start  of  each  meeting 
we  usually  posed  the  question: 

"Tell  us  about  the  most  advanced  holistic  uses  of  model-centric  engineering  you  have  seen  in  use  on 
projects/ programs  or  that  you  know  about" 

We  also  state  the  question  posed  by  our  NAVAIR  sponsors: 

Do  you  think  it  is  "technically  feasible"  for  an  organization  like  NAVAIR  to  have  a  radical 
transformation  through  model-centric  engineering  (everything  digital)  and  reduce  the  time  by  25 
percent  for  developing  and  delivering  a  major  5th  generation  air  vehicle  system? 

Most  of  the  discussions  with  industry  and  commercial  organizations  were  governed  by  some  type  of 
Proprietary  Information  Agreement  (PIA)  or  Non-disclosure  Agreement  (NDA).  In  addition,  some  of  the 
provided  material  is  marked  in  a  manner  that  limits  our  distribution  of  the  material.  Due  to  the  need  to 
sign  a  PIA/NDA,  we  are  being  careful  about  disclosing  those  organizations  in  this  report.  In  addition, 
because  we  cannot  disclose  information  about  commercial  or  industry  organizations,  we  limit  how  we 
discuss  the  other  organizations  too,  and  reference  only  published  and  publicly  available  information. 

We  have  created  over  150  pages  of  narrative  from  the  meeting  minutes,  which  generalize  the  information 
we  heard  during  the  discussions.  NAVAIR  wants  to  share  it  with  our  NAVAIR  research  team,  therefore  we 
are  including  the  following  notice  on  meeting  minutes  that  are  distributed  to  our  team,  per  Jaime 
Guerrero,  Director,  SEDIC  -  AIR-4.1,  NAVAIR: 

DISTRIBUTION  STATEMENT  D.  Distribution  authorized  to  the  Department  of  Defense  and  U.S.  DoD 
contractors  only.  Other  requests  shall  be  referred  to  SEDIC  Director,  AIR-4.1,  Naval  Air  Systems 
Command,  22347  Cedar  Point  Rd.,  Bldg.  2185/Rm.  3A54,  Patuxent  River,  MD  20670  -  (301)  342-0075. 

These  meeting  minutes  are  not  part  of  the  official  deliverable,  but  because  this  report  is  an  official 
deliverable  and  will  be  publically  available,  we  are  not  going  to  include  any  information  about  the 
organizations  that  we  met  with.  Instead,  we  provide  a  generalization  through  the  following  narrative  and 
discuss  the  results  in  the  aggregate. 

Our  team  developed  a  guideline  for  our  collective  NAVAIR  team  to  hold  discussions  in  an  effort  to 
understand  the  most  state-of-the-art  and  holistic  approaches  to  model-centric  engineering.  The  objective 
for  our  team  members  was  to  facilitate  conversations  through  discussions  that  draw  out  insights  into 
leading  advances  in  model-centric  engineer.  We  agreed  early  on  with  the  sponsor  that  open-ended 
discussions,  as  opposed  to  surveys,  would  bring  out  new  and  more  innovative  approaches  and  strategies. 
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We  were  particularly  interested  in  demonstrations  of  actual  technical  capabilities.  We  also  wanted  to 
understand  the  critical  gaps  and  limitations  too. 


Discussion  Narratives  and  Measurement  Summary 


We  created  a  collection  instrument  to  provide  a  constructive  approach  to  conduct  a  discussion  with 
organizations  as  well  as  a  way  to  provide  some  type  of  quantitative  measure  associated  with  using 
subjective  information  to  rate  the  "state-of-the-art"  of  a  holistic  approach  to  model-centric  engineering. 
We  use  a  qualitative  subjective  approach  that  computes  a  probabilistic  value  associated  with  crosscutting 
factors  associated  with  the  technical  Vision  for  this  task. 

The  collection  instrument  uses  an  Excel  spreadsheet  as  the  input  mechanism  to  collect  factor  values 
about  an  organization's  use  of  MBSE.  Each  row  in  the  spreadsheet  represents  the  subjective  information 
associated  with  one  organization.  The  latest  version  of  the  instrument  includes  one  organizational 
classifier  and  22  factors. 

As  shown  in  Figure  13,  the  model  produces  two  probability  distributions,  one  for  the  Technical  Risk  State 
of  the  Art  (max  of  10),  and  another  for  the  Technical  State  of  the  Art  (max  100),  shown  larger  in  Figure 
14.  We  think  these  factor  values  provide  a  probabilistic  value  that  is  related  to  the  technical  feasibility 
questions,  and  help  in  reflecting  on  the  factors  that  are  enablers,  as  well  as  help  identify  where  gaps  exist 
that  must  be  addressed  through  risk  identification  and  management.  We  made  some  adjustments  to  the 
factor  weightings  based  on  some  of  the  early  discussions  we  have  had  with  organizations.  We  also  used 
a  non-weighted  spreadsheet  to  illustrate  the  sensitivities  towards  what  we  consider  more  advanced 
factors  (e.g.,  metamodel  transformation). 
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Figure  13.  Measurement  Collection  Instrument 


28 


The  analysis  did  highlight  several  of  the  challenge  areas  listed  below,  but  in  the  end  it  was  not  able  to  deal 
with  the  software  complexity  issue  in  achieving  the  goal  of  a  25  percent  reduction  in  the  time  to  deliver  a 
large  scale  air  vehicle  system.  Therefore,  we  addressed  this  topic  with  the  scenarios  provided. 

Some  of  the  enablers  extracted  from  our  discussions  were  (this  list  is  not  exhaustive): 

■  Mission-level  simulations  that  are  being  integrated  with  system  simulation,  digital  assets  & 
products  providing  a  new  world  of  services 

■  Leaders  are  embracing  change  and  adapting  to  use  digital  strategies  faster  than  others 

■  Modeling  environments  to  create  dynamic  Operational  Views  (OV1)  are  more  commonly  used, 
which  used  to  be  static  pictures 

■  ID,  2D  &  3D  Models  have  Simulation  and  Analysis  Capabilities  (mostly  physics-based) 

■  Platform-based  approaches  with  virtual  integration  help  automakers  deliver  vehicle  faster 

■  Modeling  and  simulation  in  the  automotive  domain  is  reducing  the  physical  crash  testing  (e.g., 
from320  to  80,  and  400  to  40) 

o  This  could  imply  that  modeling  and  simulation  can  reduce  test  flights,  which  are  very  costly 
as  it  is  difficult  to  get  flight  clearances  on  air  craft  that  have  advanced  new  capabilities 

■  Design  optimization  and  trade  study  analysis 

■  Engineering  affordability  analysis 

■  Risk  modeling  and  analysis 

■  Pattern-based  modeling  based  on  ontologies  with  model  transformation  and  analysis 

■  Domain-specific  modeling  languages 

■  Set-based  design 

■  Modeling  and  simulation  of  manufacturing 
We  next  discussed  the  gaps  and  challenges: 

■  Model  integration,  interoperability,  and  transformation  between  domains  and  disciplines  is  a 
challenging  issue 

o  Still  mostly  stove-piped 

o  Systems  engineering  is  about  integration  of  disciplines  across  many  domains,  but  there  is 
not  a  lot  of  cross-domain  integration  in  the  simulation  capabilities  (only  a  few  exceptions) 
o  We  have  a  "sea"  of  models,  simulators,  solvers,  etc.,  but  we  do  not  have  consistent  meaning 
across  or  between  them 

o  Lack  of  precise  semantics  especially  in  both  behavior  of  models  and  timing/interactions  of 
models 

o  This  limits  the  full  spectrum  of  analyses  and  simulations  needed  to  provide  adequate 
coverage  over  a  system's  capabilities;  it  is  also  not  well  integrated  "upward"  into  the 
mission  simulations  (although  there  is  effort  to  do  this) 
o  Some  are  looking  at  how  to  work  and  integrate  a  federation  of  models  and  digital  assets,  but 
that  is  not  an  ideal  solution 

■  Many  believe  we  can  "engineer"  the  "integration"  of  models/simulations  to  address  this 
challenge 

■  Increasing  complexity  in  software,  which  is  90  percent  of  the  functionality  in  large  scale  air 
vehicle  systems 

■  Use  of  unvalidated  models 

o  Note:  unvalidated  does  not  mean  that  the  model  is  invalid 
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Task  1  -  Process 


It  was  decided  by  the  sponsor  that  to  obtain  a  uniform  perspective  on  the  feedback  from  the 
organizational  discussions  that  the  Principal  Investigator  and  lead  NAVAIR  representative  for  this  task 
would  attend  all  of  the  organizational  meetings.  After  a  meeting  with  an  organization,  we: 

■  Completed  one  row  of  the  spreadsheet 

■  Wrote  a  summary  reflecting  on  the  key  unique  capabilities  of  the  organization 
o  Most  meetings  resulted  in  five  to  10  pages  of  narrative 

The  spreadsheet  responses  are  incorporated  in  a  master  list.  The  value  for  each  factor  is  entered  in  a 
modeling  tool,  which  quantifies  the  subjective  inputs  provided  to  the  tool,  as  shown  Figure  14.  The 
maximum  value  of  the  mean  of  the  probability  distribution  is  100.  As  reflected  in  Figure  14,  it  was  decided 
that  because  there  are  some  organizations  that  require  confidentiality  or  proprietary  information 
agreements,  we  have  decided  to  keep  the  names  of  all  organizations  anonymous.  In  addition,  the 
narrative  highlights  the  most  key  capabilities  and  challenges,  but  is  generalized  to  ensure  each 
organization's  anonymity.  Additional  details  about  interpreting  the  results  are  provided  in  Section  0. 


Scenario  Collection 

After  each  discussion  we  complete  the  spreadsheet  collection  mechanism  as  shown  in  Figure  15  by 
working  through  the  row  and  uses  the  pull  down  menus  to  select  a  factor  value  of  Low,  Medium,  or  High. 
A  complete  list  of  factors  is  provided  in  a  worksheet  tab  of  the  spreadsheet  collection  mechanism  titled: 
Factor  Meaning-Definition.  Example  definitions  are  provided  in  Section  0,  with  some  additional  rationale; 
a  complete  set  of  definitions  is  provided  in  Discussion  Collection  Instrument  Guide  and  provided  in  the 
back  up  material  of  this  report.  The  objective  is  to  identify  if  there  are  state-of-the-art  methods,  tools, 
processes  and  innovative  strategies  that  are  being  used  to  significantly  advance  the  development  and 
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deployment  of  systems  through  model-centric  engineering  and  related  approaches,  and  to  incorporate 
these  concepts  in  the  Vision  model. 


Organization  Name  Organizational  Scope 


Factors  (by  Category) 


Menu  for  selecting 
factors  value  (L,  M,  H) 


Figure  15.  Spreadsheet  Instrument  Collection 


Organizational  Type 

The  general  convention  used  is: 

■  Academia  -  this  should  include  strictly  academic  organizations;  other  organizations  performing 
research  should  either  be  Industry,  Commercial  or  Government 

■  Industry  -  these  are  organizations  that  are  using  MBSE  to  develop  products  and  systems  (e.g., 
those  contractors  to  NAVAIR  that  produce  air  vehicle  systems) 

■  Commercial  -  this  is  a  special  case  of  Industry  that  relates  to  MBSE  product  developers 

o  These  organizations  either  develop  MBSE  tools  and  services,  or  may  apply  them  with 
Industry  or  Government  customers 

o  These  organizations  are  in  the  list,  because  they  may  have  insights  into  some  of  the  leading 
or  novel  uses  of  the  tools,  and  they  are  aware  of  the  need  to  continually  advance  their  own 
product  and  services 

■  Government  -  this  includes  military,  and  other  non-military  organizations  such  as  Department  of 
Transportation,  and  the  FAA 


Organizational  Scope 

One  challenge  for  some  of  the  initial  uses  of  the  collection  mechanism  was  to  appropriately  reflect  on  the 
organization  scope  for  which  these  model-centric  engineering  usage  questions  apply.  Remembering  that 
the  key  objective  of  the  research  task  is  to  assess  the  "Technical  Feasibility"  of  MCE.  We  were  pleased 
that  most  discussions  were  held  with  senior  and  knowledgeable  individuals  such  as:  such  Chief  Engineer, 
Chief  Technical  Offer,  Program  Manager  or  some  MBSE  technical  experts  in  the  organization.  To  carry  this 
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a  step  further,  it  might  also  be  important  to  keep  the  "systems"  perspective  in  mind,  because  some  of 
the  concepts  discussed  applied  to  the  hardware  level,  and  advanced  approaches  to  software  (e.g.,  the 
control  laws  for  the  F35  are  built  in  Simulink,  with  auto  code  generation,  and  a  significant  portion  of  auto 
test  generation),  but  these  types  of  approaches  may  only  be  emerging  in  use  at  the  systems  level.  We 
wanted  to  understand  how  comprehensive  the  use,  and  also  understand  the  technical  gaps. 

Finally,  this  research  is  not  limited  to  NAVAIR,  however  when  thinking  about  NAVAIR  systems  the  scope 
is  often  quite  large  and  involves  large-scale  multi-year  programs,  where  the  systems  are  actually  built  by 
one  or  more  contractors. 

Therefore,  we  factor  in  the  organizational  scope  associated  with  the  MBSE  discussion:  Program,  Project, 
an  entire  Business  Unit,  Platform  (e.g.,  a  type  of  a  specific  aircraft,  tank,  automobile),  Department,  or 
Site.  As  a  result  some  of  the  organizations  with  lower  scores  had  more  limited  discussions  and  did  not 
cover  a  broad  lifecycle  perspective  on  MCE. 


Factors  Definition  Example 

The  factor  categories  shown  in  Figure  15  do  not  necessarily  relate  to  specific  MBSE  practices,  rather  they 
are  higher-level  characteristics  of  the  organization's  ability  to  leverage  the  use  of  models  and  the 
associated  technologies  that  enable  simulations  and  accelerate  the  analyses,  design,  synthesis,  V&V  and 
manufacturing  processes.  For  example: 

■  Crossing  the  Virtual  V  is  a  high-level  category  that  has  significant  weighting  in  the  model, 

because  our  sponsor  emphasized  this  as  a  critical  need  and  the  ability  to  understand  the  design 
capabilities  through  early  V&V  activities  at  the  system  and  mission  level  (as  opposed  to  the 
subsystem  or  component  level).  This  factor  category  has  three  main  factor  characteristics: 
o  Simulation  of  Integration 

•  If  an  organization  has  simulations  of  integration  or  integrated  simulations  across  domains 
of  the  system,  and  especially  at  the  "higher"  levels  of  the  "V,"  this  is  a  likely  indicator  that 
such  an  organization  is  likely  to  have  the  ability  to  understand  simulations  of  the  system 
within  the  context  of  a  mission,  and  there  is  a  better  understanding  of  the  integration 
impacts,  because  the  simulations  are  integrated  or  represent  integration,  including 
critical  temporal  aspects  in  simulation 

•  This  includes  the  integration  of  surrogates,  use  of  instrumented  systems,  actual  system 
components,  new  prototypes,  and/or  in  development 

•  Other  attributes  of  this  type  of  simulation,  would  be  human-in-the-loop,  as  well  as  multi¬ 
level  mixed  fidelity  simulations  that  provide  the  right  abstractions  at  the  right  level 

o  Formal  analysis 

•  This  means  that  the  analysis  is  automated,  because  the  models  are  semantically  rich;  we 
are  looking  for  automated  analysis,  rather  than  looking  at  humans  performing  the  analysis 

•  Models  increasingly  have  more  semantic  richness  that  enable  automated-types  of 
analysis 

•  Models  are  increasingly  being  integrated  (see  factor  category  Cross  Domain  Coverage) 
o  Domain  specific 

•  These  types  of  systems  involve  the  integration  of  many  disciplines 

•  Models  need  to  provide  the  relevant  abstractions  that  are  related  to  the  domain  of  the 
engineer  performing  the  work 
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•  Domain-specific  modeling  is  an  emerging  type  of  modeling  that  often  provides  the 
relevant  abstractions,  with  the  semantic  richness  to  enable  automated  analysis, 
simulation,  synthesis  (generation)  and  automated  test 

•  DARPA-sponsored  research  that  demonstrated  the  capability  for  continuously  evolving 
Domain  Specific  Modeling  and  analyses  in  2008  as  an  emerging  capability  and  theme  [39], 
[99] 

•  Modeling  languages  like  System  Modeling  Language  (SysML)  are  general  purpose  [68] 
they  generally  lack  the  semantic  richness  needed  for  formal  analysis  leveraging  for 
example  formal  methods  of  automated  V&V  [21];  while  they  may  be  understood  by 
system  engineers,  control  system  engineers  would  prefer  Matlab/Simulink,  and  other 
engineers  may  require  other  domain-specific  models  and  tools  (e.g.,  computational  fluid 
dynamics,  radio-frequency,  heat  transfer).  However,  SysML  does  provide  an  underlying 
framework  for  holding  system  model  information  [127],  yet  the  models  are  not 
executable  even  with  existing  plug-in  authoring  tools  [32], 


Discussion  Summaries 

There  are  detailed  meeting  notes  that  were  shared  with  the  NAVAIR  research,  but  they  were  not  generally 
released  as  many  of  the  discussions  with  industry  and  commercial  organizations  were  governed  by  some 
type  of  Proprietary  Information  Agreements  (PIA)  or  Non-disclosure  Agreements  (NDA). 


Predictive  Model 

This  section  is  provided  for  those  interested  in  more  details  about  the  mechanism  for  converting  the 
subjective  factors  into  a  quantitative  number.  The  model  is  created  using  a  Bayesian  Network  [97]  (BN) 
tool.  There  are  three  basic  reasons  we  selected  this  approach,  BNs: 

1.  Provide  for  the  translation  of  subjective  information  into  quantitative  probabilities 

2.  Allows  for  the  use  of  subjective  expert  qualitative  judgment  and  captures  the  casual  relationship 
between  subjective  factors 

3.  Create  weighting  for  more  advanced  capabilities  such  as  meta-model  transformation;  an 
organization  that  can  demonstrates/uses  such  capabilities  is  very  advanced  in  the  use  of  models 
as  this  would  support  an  objective  of  cross-domain  integration 

The  outputs  are  also  probability  distributions,  which  means  that  they  provide  some  type  of  range  to 
provide  a  comparison  between  different  organizations.  The  specific  numbers  are  not  necessarily  as 
important  as  our  ability  to  compare  different  organizations  and  relate  the  responses  back  to  advanced 
uses  of  MBSE  and  related  enabling  technologies.  While  no  organization  may  have  all  "High"  values,  this 
approach  provides  a  way  to  look  at  relative  comparison  in  conjunction  with  the  narratives.  Each  of  the 
nodes  in  the  BN  shown  in  Figure  16  provides  a  type  of  weight  called  a  conditional  probability.  We  have 
used  the  team's  judgment  to  weight  the  different  nodes  in  a  way  that  would  relate  to  evaluating  the  key 
question  for  this  task:  is  it  technically  feasible  to  "do  everything  with  model."  In  addition,  we  will  refine 
the  weightings  as  we  proceed  through  discussions. 
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Figure  16.  Bayesian  Network  Underlying  Collection  Instrument 


Rationale  for  Bayesian  Networks 

A  Bayesian  network  is  a  representation,  which  organizes  one's  knowledge  about  a  particular  situation 
into  a  coherent  whole  [40],  They  are  increasingly  being  used  in  the  modeling  of  uncertain  and  incomplete 
knowledge.  Bayesian  thinking  is  inherently  more  intuitive  than  many  other  evaluation  techniques;  it  best 
reflects  commonsense  thinking  about  uncertainty  that  humans  have.  We  frequently  use  words  like 
"likely,"  "rarely,"  and  "always"  to  express  varying  degrees  of  uncertainty.  Subjective  probability  is  our 
way  of  assigning  numbers  (between  0  and  1)  to  these  different  degrees  of  uncertainty,  and  our 
probabilities  can  change  as  we  are  presented  with  new  information,  or  we  have  new  experiences  which 
cause  a  shift  in  beliefs  or  expectations.  When  this  shift  occurs,  the  way  our  probabilities  change  are 
governed  by  Bayes'  rule. 

A  Bayesian  network,  as  used  in  this  framework,  is  a  joint  probability  distribution  and  as  such,  any  question 
that  can  be  asked  in  a  probabilistic  form  can  be  answered  with  a  stated  level  of  confidence.  Some  typical 
questions  might  be: 

■  Given  a  set  of  effects,  what  are  the  causes? 

■  How  can  an  outcome  be  controlled,  given  a  set  of  circumstantial  values? 

■  If  we  model  a  causal  relationship,  what  result  would  an  intervention  or  change  bring? 

While  there  are  several  ways  to  structure  a  Bayesian  network,  we  used  prior  experience  to  structure  the 
model.  The  subjective  factors  in  the  spreadsheet  instrument  map  directly  to  the  yellow  oval  nodes  of  the 
BN  model.  The  purple  rectangles  are  intermediate  nodes  and  generally  relate  to  factor  categories.  The 
orange  rectangles  represent  the  probability  outputs  of  both  Technical  state  of  the  art  (Task  3)  and  the 
Technical  Risk  state  of  the  art  (Task  4). 
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Data  -  Likert  Scales  (Ranked  Scales) 

The  subjective  factors  in  the  model  use  a  Ranked  node  type,  which  is  a  type  of  Likert  Scale.  It  is  important 
to  note  that  although  Likert  scales  are  arbitrary,  they  can  retain  a  level  of  reliability  for  our  use.  The  value 
assigned  to  a  Likert  item  has  no  objective  numerical  basis,  either  in  terms  of  measure  theory  or  scale 
(from  which  a  distance  metric  can  be  determined).  In  this  case,  the  value  assigned  to  a  Likert  item  has 
been  determined  by  the  researcher  constructing  the  Bayesian  network,  but  can  be  refined  as  the  research 
progresses.  The  results  have  been  a  balanced  representation  of  strata  and  detail. 

Typically,  Likert  items  tend  to  be  assigned  progressive  positive  integer  values.  Likert  scales  usually  range 
from  2  to  10  -  with  5  or  7  being  the  most  common.  In  this  model,  3  levels  are  used,  at  least  for  now  as  it 
minimizes  the  number  of  computational  states,  which  minimizes  time  for  the  analysis.  The  progressive 
structure  of  a  Likert  scale  is  such  that  each  successive  Likert  item  is  treated  as  indicating  a  'better' 
response  than  the  preceding  value.  Note  that  the  direction  of  'better'  (i.e.,  Higher)  depends  on  the 
wording  of  the  factor  definition. 

In  terms  of  good  practice,  a  bias  in  the  computations  may  result  if  the  suppliers  of  data  for  the  framework 
do  not  agree  on  the  relative  values  of  each  factor.  However,  there  are  enough  factors  that  a  bias  in  one 
or  two  values  will  likely  not  skew  the  results  significantly. 

Task  2  -  Common  Model  Lexicon 


The  team  was  tasked  at  the  kickoff  meeting  to  create  a  common  lexicon  for  things  related  to  "models"  in 
the  systems  engineering  domain,  and  in  fact,  in  the  broader  engineering  space.  An  example  of  this  is  what 
is  meant  by  the  word  "model."  This  lexicon  is  focused  on  providing  a  common  language  for  all  to  use  in 
the  development  and  evolution  of  things  related  to  models. 

Most  engineers  will  agree  that  a  model  is  a  facsimile  of  reality.  Yet,  to  an  industrial  engineer,  a  model 
may  represent  a  production  facility;  to  a  mechanical  engineer  it  may  be  a  finite  element  model  analysis; 
to  a  systems  engineer  it  may  be  an  IDEFO  [65]  or  a  SysML  representation  of  the  system,  subsystem,  or 
some  lower  level  element.  None  of  those  perspectives  are  wrong;  they  are  just  different  views  of  some 
part  of  the  same  enterprise. 

Some  claim  that  there  is  no  existing  model  lexicon  or  taxonomy  [10],  although  there  are  a  number  of 
different  types  of  taxonomies  that  all  fit  within  the  more  general  context  of  a  model  lexicon  [36],  [127], 
The  Object  Management  Group  (OMG)  in  conjunction  with  INCOSE  has  established  an  Ontology  Action 
Team  to  work  on  similar  efforts  [90],  The  NDIA  Modeling  &  Simulation  Committee  is  about  to  approve 
the  Final  Report  on  the  Identification  of  Modeling  and  Simulation  Capabilities  by  Acquisition  Life  Cycle 
Phase  [11], 

Status:  we  have  captured  over  750  named  lexicon  items  related  to  the  term  "model,"  including  levels, 
types,  uses,  representations,  standards,  etc.  We  have  had  several  reviews  and  it  has  been  refined  and 
restructured  several  times.  The  reviewers  stated  that  the  graphical  representation  that  was 
automatically  generated  from  the  lexicon  was  useful. 


Ontology  vs.  Lexicon 

According  to  Wikipedia,  ontologies  are  the  structural  frameworks  for  organizing  information  and  are  used 
in  artificial  intelligence,  the  Semantic  Web,  systems  engineering,  software  engineering,  biomedical 
informatics,  library  science,  enterprise  bookmarking,  and  information  architecture  as  a  form  of 
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knowledge  representation  about  the  world  or  some  part  of  it  [122],  The  creation  of  domain  ontologies  is 
also  fundamental  to  the  definition  and  use  of  an  enterprise  architecture  framework. 

A  lexicon  is  a  similar  concept  -  it  is  normally  a  book  or  glossary  like  document,  or  words  (and  their 
definitions)  in  a  language  or  domain,  arranged  in  alphabetical  order.  The  team  decided  that  a  simple 
glossary  would  not  be  sufficient  because  it  does  not  show  the  relationships  between  terms. 

In  simplistic  terms,  an  ontology  becomes  a  complex  network  of  words,  and  their  relationships  to  each 
other.  A  lexicon  is  a  glossary.  Neither  was  exactly  what  was  needed  for  this  project.  The  team  needs 
something  that  provides  definitions  and  simple  relationships  to  related  terms  -  not  complex,  rigid 
definitions.  We  chose  to  use  the  word  Lexicon,  though  the  words  could  also  be  represented  in  a  tree-like 
structure  that  is  common  for  ontologies. 


Tool  for  Representing  Word  Relationships 

There  are  tools  available  for  creating  ontologies.  There  actually  exists  a  class  of  workers  that  consider 
themselves  Ontologists.  These  tools  come  in  many  different  flavors  -  from  open  source  tools  to 
commercial  tools.  The  common  thread  is  that  they  create  graphical  representations  as  shown  in  an 
example  in  Figure  17.  These  tools  require  rather  rigorous  definitions  and  relationships  to  complete.  The 
open  source  tools  are  actually  very  good,  and  very  robust.  However,  after  some  evaluation  of  available 
open  source  tools,  the  team  decided  that  it  would  be  better  to  create  a  straightforward  spreadsheet  of 
terms  (e.g.  a  Lexicon),  and  then  create  a  script  that  could  represent  that  lexicon  graphically. 

peptide 


Figure  17.  Sample  Graphic  Representation  from  Ontological  Software 


The  Lexicon 

A  spreadsheet  was  first  created  in  Excel.  At  first,  the  team  was  simply  capturing  the  words,  their 
definition,  and  where  it  made  sense,  a  key  reference  or  two  for  that  definition.  Table  1  shows  the 
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implementation  of  this  data  gathering  spreadsheet.  Once  the  decision  was  made  to  create  a  tool  to  make 
this  information  available  graphically,  and  also  on  the  web,  it  became  apparent  that  a  "relationship"  data 
element  was  necessary.  Therefore,  the  data  collection  tool  captures: 

■  Name 

■  Has  Parents  [0  or  more]  separate  with  if  more  than  one 

■  Definition 

■  Sample  Usage 

■  Also  Known  As 

■  Key  URL  (optional) 

The  current  spreadsheet  represents  a  continuous  accumulation  of  relevant  terms,  their  definitions,  and 
their  classification.  The  initial  definitions  have  been  drawn  from  readily  available  sources  on  the  Internet 
(often  from  Wikipedia  where  the  assumption  is  that  it  has  been  created  by  a  group  of  people  with  both 
knowledge  and  passion  about  the  subject).  In  other  cases  members  of  the  research  team  have  authored 
a  definition  based  on  their  understanding  of  the  term  in  a  relevant  context. 

Table  1.  Initial  Lexicon  Capture  Tool 


Name 

Parents 

Definition 

in  mi . . . 

Also  Known  As 

Key  URL  or  Reference 

Metadata 

Definition 

Searchable  information  describing  the  characteristics 
of  data;  data  or  information  about  data;  or 
descriptive  information  about  an  object's  data,  data 
activities,  systems,  and  holdings. 

Metadata  are  traditionally  found  in  the  card  catalogs 
of  libraries.  As  information  has  become  increasingly 
digital,  metadata  are  also  used  to  describe  digital  data 
using  metadata  standards  specific  to  a  particular 
discipline.  By  describing  the  contents  and  context  of 
data  files,  the  quality  of  the  original  data/files  is 
greatly  increased.  For  example,  a  webpage  may 
include  metadata  specifying  what  language  it  is 
written  in,  what  tools  were  used  to  create  it,  and 
where  to  go  for  more  on  the  subject,  allowing 
browsers  to  automatically  improve  the  experience  of 

users. 

DoD  M&S  Glossary.  2011. 
http://www.acqnotes.com/Att 
achments/DoD%20M&S%20GI 
ossary%201%200ct%2011.pdf 

Metamodel 

Model  Types 

A  model  of  a  model.  Metamodels  are  abstractions  of 
the  M&5  being  developed  that  use  functional 
decomposition  to  show  relationships,  paths  of  data 
and  algonthms,  ordering  and  interactions  between 
model  components  and  subcomponents. 

The  Object  Management  Group  (OMG)  has  developed 
a  metamodeling  architecture  to  define  the  Unified 
Modeling  Language  (UML),  called  the  Meta-Object 
Facility 

surrogate  model  or  response 
surface  models  (via 
http  ://sumowiki .  intec.  ugent.be 
/Main  3age) 

DoD  M&S  Glossary.  2011. 
http://www.acqnotes.com/All 
achments/DoD%20M&S%20GI 
ossary%201%200ct%2011.pdf 

Method 

Definition 

A  particular  form  of  procedure  for  accomplishing  or 
approaching  something,  especially  a  systematic  or 
established  one. 

https://www.google.com/Pgw 

s_rd=ssllfq=define+method 

Metric 

Definition 

A  measure  of  the  extent  or  degree  to  which  a 
product  possesses  and  exhibits  a  certain  quality, 
property,  or  attribute.  (SISO-REF-002-1999) 

http  ://m  sco.  mil/MSGIossary_T 
RM_  M.html 

Ministry  of  Defence  (UK) 
Architecture  Framework 
(MODAF) 

Framework 

Ministry  of  Defense  (UK)  Architecture  Framework 

https://www.gov.uk/mod- 
architecture  framework 

Mission 

Scenario 

Is  the  top-level  function  of  the  system;  the  one  that 
synthesizes  all  transformation  of  all  inputs  and 
solicitations  into  outputs  and  reactions.  (Created  for 
SEBoK) 

Simulations  tends  to  be  broadest  in  the  national 
security  Analysis  Department,  where  multiwarfare 
analysis  requires  a  variety  of  such  simulations,  but 
mission-level  Simula-  tions  for  specific  mission  areas 
are  also  employed  in  several  other  departments 

Sortie 

http://sebokwiki  .org/wi  ki/Miss 
ion_(glossary) 

Intuitively,  many  of  the  terms  in  this  spreadsheet  are  ambiguous  and  their  meaning  is  highly  dependent 
on  the  context  and  usage  domain.  This  has  been  found  to  be  true  in  reality  also  as  terms  are  collected 
from  various  domains.  It  is  therefore  important  to  emphasize  that  this  is  an  evolving  process. 


Sources  of  Information 

There  were  a  number  of  sources  used  for  the  initial  Lexicon.  Journal  papers  on  MBSE  provided  a  good 
first  cut.  Interestingly,  an  article  from  the  Journal  of  Object  Technology  [50]  proved  to  be  very  useful. 
Other  sources  included  The  Open  Group,  the  Object  Management  Group,  INCOSE,  NDIA,  and  Wikipedia. 
Other  contributors  include: 

■  NAVAIR 

■  OSD(AT&L)/OASD(R&E)/ODASD(SE)/EE 

■  USAFAFMCAFRL 

■  Systems  Modeling  &  Simulation  Working  Group  (SMSWG) 
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■  NAFEMS 


Web  Presentation 

A  short  script  was  created  that  takes  the  information  contained  in  the  data-entry  spreadsheet,  and 
generates  web-based  artifacts.  Figure  2  shows  the  published  page  as  it  looks  at  the  time  of  this  report12. 
This  page  includes  four  sections: 

■  Model  Lexicon  Overview  (Figure  18) 

■  Model  Representation/Lexicon  (Figure  20) 

o  This  is  a  generated  image  produced  by  vizGraph,  but  with  over  750  lexicon  items 
o  For  the  reviews  it  was  printed  out  at  about  8  feet  by  37  feet 

■  Hyperlinked  Tree  of  the  Model  Lexicon  (Figure  19) 

o  As  an  alternative,  a  collapsible  and  expandable  tree  (outline)  allows  people  to  understand 
the  hierarchy  of  model  lexicon  with  hyperlinks  to  a  particular  lexicon  definition. 

■  Definitions  -  A  common  structure  is  used  for  each  term  (Figure  21) 


12  The  final  location  of  the  lexicon  may  move  to  another  location. 
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Figure  18.  Published  Web  Page  from  Data  Collection  Spreadsheet 
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Model  Representation 

There  is  a  graphical  representation  of  the  lexicon  generated  by  vizGraph. 

Click  here  image. 

Note:  this  is  a  large  image  and  make  take  a  few  seconds  to  load. 

model_lexicon 

model_lexicon  Tree 

Model  lexicon 

▼  Modeling 

▼  Model 

Life  cycle  model  management 

►  Model  representations 
Model  technique 
Model  transformation 

►  Model  types 

►  Model  uses 

►  Modeling  approach 

►  Modeling  standards 
Representation 

►  Modeling  &  simulation 

►  Simulation 

▼  Modeling  community  of  interest  (coil 

►  Business 

▼  F.npincerinp 

Capability  maturity  model  integration  (emmil 

Concurrent  engineering 
Control  systems  engineering 

►  Design 

►  Requirements  engineering 
Specialization 

Specialty  engineering 

►  Systems  engineering 
►  Modeling  terms 


Figure  19.  Model  Representation  and  Lexicon  Tree 
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The  definitions  table  shown  in  Figure  21,  is  a  screen  image  from  the  website,  and  includes  the  following 
columns: 

■  Name 

■  Definition 

■  Parent 

o  This  is  a  hyperlink  to  the  parent  in  the  table 

■  Tree 

o  This  is  a  hyperlink  back  to  the  collapsible  and  expandable  tree  (outline);  clicking  on  this 
hyperlink  takes  the  focus  back  to  the  name  in  the  tree  only  if  the  item  is  expanded  in  the 
tree 

■  Sample  Use 

■  Key  Source  (if  applicable) 
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Model  Lexicon  Definitions 


Name 

Definition 

Parent 

Tree 

Sample  Use 

Source 

2d  modeling: 

a  geometric  model  of  an  object 
as  two-dimensional  figure, 
usually  on  the  Euclidean  or 
Cartesian  plane 

mechanical 

modclinc 

mechanical 
modclinc.  2d 
modeling 

3d  solid  model: 

the  product  of  3D  solid 
modeling 

mechanical 

modeling 

mechanical 
modclinc.  3d 
solid  model 

3d  solid 
modeling: 

is  the  process  of  developing  a 
mathematical  representation  of 
any  three-dimensional  surface 
of  object  (cither  inanimate  or 
living)  via  specialized  software 

mechanical 

modclinc 

mechanical 
modclinc.  3d 
solid  modclinc 

i/lore  Sampl 

L  i 

3 

Model  driven 
architecture: 

a  software  design  approach  for 
the  development  of  software 
systems 

model 

tcchniauc 

model 

tcchniauc. 
model  driven 

architecture 

\7 

httD:./Avww.omc.ore/mda 

Model  levels: 

Level  of  the  system  or  system 
of  systems;  Also  discussed  in 
terms  of  Resolution  level.  The 
amount  of  detail  or  degree  of 
aggregation  employed  in  the 
model  or  simulation 

model  lexicon 

model  lexicon. 

model  levels 

Often  used  in  modeling 
and  simulation  world  to 
discuss  the  differen  types 
of  models 

Model  lexicon: 

the  words  used  in  a  language  or 
by  a  person  or  group  of  people 

model  lexicon 

This  is  a  lexicon 
associated  with  terms 
derived  from  models  and 
modeling 

Model 

management: 

Approaches  to  managing 
models 

model  lexicon 

model  lexicon. 

model 

manaecmcnt 

Configuration  control, 
PLM,  CATIA 

Figure  21.  Tabular  Representation  of  Lexicon 


Task  3  -  Modeling  the  Vision  and  Relating  to  the  "As  Is"  and  Airworthiness  Process 


Task  3  investigates  how  to  model  the  "Vision,"  "As  Is"  and  Airworthiness  process.  This  was  a  joint  effort 
with: 


■  "As  Is"  process  model  being  worked  by  Ron  Carlson  and  Paul  Montgomery,  from  Naval 
Postgraduate  School  (NPS),  and  later  led  by  Richard  Yates  from  MITRE 

■  Airworthiness  aspects  being  worked  by  Richard  Yates  and  Rick  Price  from  MITRE,  and  Brian 
Nolan  from  SOLUTE 

■  Vision  being  led  by  Mark  Blackburn  with  contributions  by  our  collaborators  and  many  others 
contributor  during  our  working  session 

The  remainder  of  this  section  provides  additional  details,  which  is  related  to  the  research  investigation, 
working  sessions,  and  task  scoping  and  refinement  as  it  has  evolved  during  this  phases  of  RT-48/118/141. 
This  sections  is  organized  as  follows: 

■  Status  of  "As  Is"  and  Airworthiness  Process  Modeling 

■  "Vision"  model  context 
o  Containing  system 
o  Designing  system 

■  Operational  Perspectives  on  a  Vision  Concept 

■  Scoping  the  boundaries  and  interfaces  between  the  Program  of  Record  and  Mission  Analysis 

■  System  Engineering  Technical  Review  (SETR)  Manager 

■  Modeling  and  tools  for  the  Vision 

■  Model-centric  engineering  perspectives  derived  from  research  and  discussions 
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o  For  example,  how  model-centric  tool  chains  can  subsume  process 
o  Crossing  the  Virtual  V 
o  Perspective  on  Integrated  MCE  Environment 


Status  of  "As  Is"  and  Airworthiness  Process  Modeling 

This  section  provides  a  brief  summary  of  the  "As  Is"  and  Airworthiness  process  modeling.  Historical  details 
on  this  effort  can  be  found  in  the  RT-48  technical  report  [19]  and  RT-118  technical  report  [22],  The  "As 
Is"  and  Airworthiness  modeling  results  are  being  documented  in  a  separate  non-SERC  report.  The  MITRE 
team  leads  this  effort  and  a  final  report  is  expected  in  December  2015. 

The  general  motivation  for  defining  the  "As  Is"  process  and  model  is  to  assure  the  stakeholders  that  the 
new  process  covers  everything  that  was  required  of  the  old  process.  In  addition,  it  provides  a  type  of  map 
to  those  "As  Is"  process  activities  and  artifacts  that  may  be  replaced  or  subsumed  through  new  methods 
and  automation. 

The  Airworthiness  process  is  used  to  ensure  that  the  necessary  evidence  is  provided  in  order  to  get  a 
flight  clearance.  Brian  Nolan  from  Solute  is  working  with  Richard  Yates  from  MITRE  to  create  a  model  for 
this  process.  They  have  used  authoritative  sources  [70]  [74],  however,  a  significant  amount  of  guidance 
is  obtained  through  interactions  with  Airworthiness  subject  matter  experts  (SME). 

There  have  been  a  number  of  attempts  and  approaches  applied  to  characterize  the  "As  Is"  process  and 
Airworthiness  constraints.  The  effort  first  started  by  identifying  the  artifacts  that  are  collected  to  support 
the  NAVAIR  System  Engineering  Technical  Review  (SETR)  process.  The  team  categorized  about  330 
artifacts  (primarily  documents),  which  were  represented  in  a  CORE  model  [118].  The  artifact  analysis 
resulted  in  a  somewhat  abstract  understanding  of  the  "As  Is"  process.  Many  of  the  artifacts  are  just 
named  items  with  no  formalized  definition  of  the  artifact's  details  or  resources  used  to  produce  them; 
SMEs  knew  the  key  information.  This  information  was  needed  to  understand  the  information  that  may 
need  to  be  captured  in  a  digital  engineering  approach. 

This  effort  also  worked  to  extract  knowledge  from  stakeholders  that  have  used  the  processes  to  further 
refine  both  the  artifacts  and  overlay  a  process.  They  developed  a  process  model  using  a  standard  SysML 
compliant  Activity  diagrams  representing  the  actions  of  three  NAVAIR  competencies  (Software 
Engineering,  Propulsion  and  Power,  and  Human  Systems)  for  SETR  events,  with  a  focus  on  how 
Airworthiness  issues  are  identified  and  managed.  The  high-level  findings  include: 

■  Current  NAVAIR  engineering  practices  are  documented  at  the  "What"  level  but  not  the  "How" 
o  NAVAIR  practices  "good  system  engineering"  and  utilizes  best-practice  and  lessons  learned 
o  "How"  NAVAIR  applies  systems  engineering  is  highly  adaptive  and  fluid  and  often  depends 

on  known/unknown  variables 

o  A  highly  prescriptive  approach  to  NAVAIR  engineering  processes  is  not  desirable  and  would 
probably  not  improve  effectiveness 

■  Airworthiness  is  not  a  process  but  a  set  of  constraints  applied  to  the  desired  capability  and  the 
proposed  engineering  design/solution 

o  Airworthiness  is  one  of  several  constraints  that  must  be  considered  when  developing 
engineering  solutions 

o  How  and  the  degree  to  which  constraints  are  met  is  managed  through  a  risk  management 
process 

o  This  is  how  a  balance  between  meeting  requirements  and  complying  with  constraints  is 
achieved 
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Recommendation  from  the  "As  Is"  process  modeling  effort: 

■  How  we  develop  NAVAIR  "As-ls"  and  "To-Be"  process  models  should  be  adapted  to  match  the 
nature  of  NAVAIR  work 

■  Some  processes  should  remain  adaptive  and  fluid  to  be  effective,  while  others  such  as  Risk  could 
be  made  more  prescriptive  and  rigorous 

■  In  some  cases  the  focus  should  be  on  the  result  of  the  process,  the  data,  and  not  the  process  - 
this  is  inherently  what  MCE  would  do 

■  Knowing  who  produces  what,  when  and  describing  the  data  in  a  common  language  may  be 
more  valuable  than  knowing  "How" 


What  is  a  model? 

We  have  heard  from  our  stakeholders  that  some  people  may  not  understand  what  is  meant  by  the  term 
model,  as  well  as  having  a  consistent  view  on  model-centric  engineering.  We  are  going  to  provide  some 
details  before  moving  on  to  the  concepts  involved  in  the  Vision. 

Modeling,  in  the  broadest  sense,  is  the  cost-effective  use  of  something  in  place  of  something  else  for 
some  cognitive  purpose.  It  allows  us  to  use  something  that  is  simpler,  safer  or  cheaper  than  reality 
instead  of  reality  for  some  purpose.  A  model  represents  reality  for  the  given  purpose;  the  model  is  an 
abstraction  of  reality  in  the  sense  that  it  cannot  represent  all  aspects  of  reality.  This  allows  us  to  deal 
with  the  world  in  a  simplified  manner,  avoiding  the  complexity,  danger  and  irreversibility  of  reality 
[107], 

George  E.P.  Box,  said:  "Essentially,  all  models  are  wrong,  some  models  are  useful."  [28] 

One  key  aspect  of  models  and  modeling  is  abstraction,  which  supports  communication  through  different 
views  with  various  levels  of  details.  Details  of  importance  can  be  emphasized  while  other  details  are  not 
described.  Most  of  us  have  been  exposed  to  models  for  a  long  time,  for  example,  a  mobile  of  the  solar 
system,  as  shown  in  Figure  22  shows  the  number  of  planets  and  might  show  the  relative  position  of  the 
planets,  but  it  does  not  accurately  show  the  plant's  size  or  distance  from  the  sun.  Different  views  can 
provide  alternative  and  relevant  perspectives  on  the  planets  of  the  solar  system  and  emphasize  the 
relative  size  of  the  planets.  To  get  an  accurate  perspective  of  a  problem  or  solution  often  requires  several 
views  with  some  type  of  formal  description  of  the  relationship  between  the  views.  For  example,  the 
distance  from  the  sun  to  each  planet  needs  to  be  described  using  consistent  units  (e.g.,  miles). 
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Figure  22.  Two  Model  Views:  Mobile  and  Relative  Size  of  Planets13 

Model-centric  capabilities  to  achieve  NAVAIR's  vision  will  heavily  rely 
on  computationally-based  models.  We  need  to  use  the  relevant 
abstractions  that  help  people  focus  on  key  details  of  a  complex 
problem  or  solution  combined  with  automation  to  support  the 
simulation  and  dynamic  analysis  of  both  the  problem  and  solution, 
along  with  the  mechanism  for  combining  the  information  collected 
from  the  various  abstractions  to  construct  a  system  correctly.  Some 
of  the  key  abstractions  can  be  categorized  into  types,  such  as: 

■  Structure  -  ID,  2D,  3D  models,  systems,  subsystems, 
components,  modules,  classes,  and  interfaces  (inputs  and  outputs) 

■  Behavior  (functionality) 

■  Timing  (concurrency,  interaction) 

■  Resources  (environment) 

■  Metamodels  (models  about  models) 

Many  of  these  model-centric  abstraction  concepts  have  existed  and  evolved  from  programming 
languages,  but  within  a  programming  language  the  combination  of  these  views  may  be  lumped  or  tangled 
together  (e.g.,  spaghetti  code).  Most  dynamic  model  capabilities  cannot  effectively  leverage  simulation, 
analysis  or  generation  capabilities  if  the  models  are  constructed  in  an  ad  hoc  way. 

Modeling  methodologies  (beyond  process  steps)  are  needed  to  guide  the  structuring  of  models  to  provide 
a  means  of  systematically  separating  these  views,  because  certain  types  of  models  are  constrained  to 
permit  only  certain  types  of  information.  Model-centric  automation  relies  on  automated  means  for 
analyzing  the  views,  deriving  information  from  one-or-more  views,  and  ultimately  pulling  sets  of  views 
together  correctly  to  produce  some  type  of  computationally-based  system,  simulation  or  associated 
analysis  artifacts  and  evidence. 


Models 


Structure/Interfaces 


Behavior  (functions) 


Concurrency 


Resources/Environment 


13  Image  credit:  www.thisisauto.com/.../wa07005i/l/ID_mobilel.jpg, 

http://elronsviewfromtheedge.wordpress.com/2006/08/23/and-you-thought-size-mattered/ 
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Vision  Model  Context 


At  the  RT-48  kickoff  meeting,  Dave  Cohen  presented  a  Vision  concept,  and  the  question  was  asked  "is  it 
technically  feasible  to  model  everything?"  As  a  result,  we  said  that  we  would  attempt  to  model  the 
"Vision."  This  term  "Vision"  was  later  clarified  as  a  future  state14  with  a  10-year  time  horizon.  The  Vision 
aligned  with  Dave  Cohen's  Vision  concept  as  well  as  the  most  advanced  MCE  approaches  that  would 
achieve  a  radical  transformation  and  significantly  reduce  the  time  from  concept  to  operations.  In  the 
remainder  of  this  section  we  still  use  the  word  "Vision"  to  characterize  the  10-year  concept. 

Two  things  have  resulted  from  our  efforts  in  researching  what  a  "Vision"  model  might  be,  and  how  it 
might  be  represented: 

1.  We  have  seen  only  a  few  examples  fragments  of  Vision-like  models  [8],  Organizations  typically 
model  only  the  systems  they  want  to  develop  and  evolve.  Often  organizations  do  not  think  about 
modeling  all  of  the  elements  of  the  environment,  referred  to  by  Ackoff  as  the  containing  system 
[1],  the  elements  and  interactions  of  the  designing  system,  including  elements  of  existing  system, 
subsystem  and  parts,  in  order  to  create  an  instance  of  the  target  system,  which  would  then  be 
stored  in  a  version  of  what  some  call  a  "System  Model"  [128]  (single  source  of  technical  truth) 

2.  One  organization  has  created  at  least  a  start  of  something  that  relates  to  the  Vision  [8];  they  use 
the  System  Modeling  Language  (SysML)  [92],  We  started  an  example  SysML  model  to  illustrate 
this  concept  too.  However,  in  some  of  our  working  sessions,  we  found  that  not  everyone  was 
familiar  or  comfortable  with  using  SysML  modeling  views. 

Therefore,  we  will  provide  some  examples  to  help  with  clarifying  the  concept  of  what  we  think  should  be 
included  in  the  Vision,  and  limit  the  use  of  modeling  notations. 

We  typically  hear  about  a  System  Model  [8],  [128],  which  should  ideally  represent  all  the  data  and 
information  to  produce  the  target  system,  as  well  as  all  of  the  evidence  that  characterizes  the  consistency, 
completeness,  and  correctness  of  the  information.  During  RT-141,  NAVAIR  began  to  characterize  this  as 
the  Single  Source  of  Technical  Truth.  In  this  case,  it  is  scoped  to  the  Program  of  Record  (POR)  for  our  task. 
The  information  to  cover  what  we  believe  to  be  the  Vision  should  include: 

o  Sufficient  information  about  the  containing  system  [1] 

o  This  information  should  come  from  the  mission  analysis  as  sets  of  desired  operational 
capabilities  with  performance  objectives 

o  NAVAIR  is  conducting  some  similar  type  of  research  effort  at  the  mission-level 
■  All  the  information  about  the  designing  system 

o  Every  tool  for  model  creation,  storage,  simulation,  analysis  and  their  interconnections  that  is 
used  to  create,  simulation,  or  produce  analysis  information  related  to  decisions  or 
dependent  information 

o  One  organization  develops  the  enterprise,  which  is  the  system  for  producing  the  target 
system 

•  This  is  often  thought  of  as  the  Information  Technology  (IT)  group 

•  NASA/JPL  integrates  and  aligns  the  development  of  these  capabilities  with  expertise  that 
is  used  on  the  program  [35] 


14  We  have  discussed  this  term  with  our  sponsor  who  also  used  the  words  End  State,  and  we  stated  that  there 
really  is  no  End  State  as  a  "Vision"  system  would  continue  to  evolve  as  the  technologies  continue  to  evolve. 
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■  All  other  platform  related  information  that  provides  some  aspects  of  the  interrelated 
capabilities  associated  with  the  POR  (system  instance  to  be  designed/evolved),  including 
revisions,  variants,  and  even  trade  spaces  analysis  results  from  design  alternatives  not  selected 

Some  of  these  perspectives  are  provided  in  Figure  2315.  This  figure  puts  into  perspective  the  mission 
capability  threads  that  have  relationships  to  different  PORs  for  the  different  platforms,  and  puts  into 
context  some  of  the  aspects  of  the  containing  system  (i.e.,  the  interrelationship  to  other  PORs  in  order  to 
support  a  capability).  This  image  abstractly  reflects  on  the  information  about  the  existing  assets  of 
previous  systems  (platforms)  that  can  play  a  role  in  model-centric  engineering: 

■  The  "assets  of  previous/existing  systems"  would  be  represented  in  some  type  of  reference 
model  or  master  template16 

o  All  of  the  existing  elements  (components)  that  could  be  put  into  a  system  derived  from 
historical  knowledge  and  reuse 

o  The  relationships  (valid  connections)  as  semantically  rich  links 
o  Model  representation  of  new  elements  (components)  from  new  design  ideas  and/or 
technological  advances 

o  Figure  8  notional  represents  this  concept  for  a  vehicle 

■  These  types  of  assets  provide  the  building  blocks  for  defining  and  refining  a  new  capability 

o  For  example,  existing  components  (sensors,  communication)  could  be  used  as  surrogate 
during  early  evaluation  and  upgraded  as  new  technology  is  developed  and  refined 
o  This  concept  is  consistent  with  what  was  discussed  by  the  automotive  industry  and  in  a 
more  advanced  way  what  was  developed  by  the  DARPA  META  project  [7] 

■  Currently,  this  information  is  largely  defined  in  documents;  it  may  be  partially  modeled,  and/or 
held  by  contractors  (data,  models,  knowledge) 

Therefore,  in  order  to  realize  the  Vision,  NAVAIR  going  forward  needs  total  and  continuous  access  to  this 
type  of  information  in  a  single  source  of  technical  truth. 


15  These  figures  come  from  a  briefing  given  by  Jaime  Guerrero  that  is  approved  for  public  release;  distribution  is 
unlimited. 

16  A  commonly  used  term  is  reference  architecture.  A  reference  architecture  in  the  field  of  software  architecture  or 
enterprise  architecture  provides  a  template  solution  for  an  architecture  for  a  particular  domain. 

Instance:  An  instance  is  a  specific  system  that  can  be  developed  using  the  template  of  the  reference  architecture. 
Any  new  technology  advances  should  be  incorporated  back  into  the  reference  architecture  for  reuse  in  the  future. 
Variance:  Items  that  meet  the  same  definition  in  the  reference  architecture  but  have  different  solutions. 
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Figure  23.  Putting  the  Vision  into  Context 


Containing  System 

The  Containing  System  must  represent  the  SoS,  including  environment  and  resources  with  sufficient 
fidelity  and  semantic  precision  to  understand  how  a  target  system  interacts  within  its  environment.  These 
types  of  views  are  needed  to  understand  the  problem  and  to  investigate,  through  models,  the  different 
alternative  systems  that  can  address  the  problem.  In  general,  a  complete  high-fidelity  representation  is 
not  possible,  therefore  there  will  be  some  type  of  abstraction  of  the  containing  system  such  as  reflected 
in  Figure  24.  This  is  one  scenario  of  a  capability,  and  the  particulars  of  the  interface  parts  can  include  the 
environment,  such  as  the  ship,  landing  surface,  arresting  system,  weapons,  weather,  threat  types, 
operators,  etc. 
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Figure  24.  Perspectives  on  Aspects  of  the  Containing  System17 


Some  of  the  current  approaches  to  modeling  used  by  NAVAIR  such  as  DoDAF  models,  are  static  and  do 
not  capture  enough  information  to  support  simulations.  They  are  semantically  imprecise  when  it  comes 
to  representing  behavior  and  the  temporal  (timing)  interaction  to  be  able  to  assess  and  predict  the 
needed  performance.  Our  field  visits  to  commercial  and  industry  organizations  reflect  on  modeling  and 
simulations  capabilities  that  are  neither  well  integrated  nor  interoperable,  but  some  organizations  are 
integrating  mission  simulations  with  system  products,  components  or  other  simulations.  While  the 
interest  and  intent  exists,  the  standards  do  not  keep  pace  with  the  technologies. 

We  acknowledge  that  there  are  limitations  in  cross-domain  model  interoperability,  consistency,  and 
limitations  transforming  models  with  the  required  semantic  precision  to  provide  accurate  information  for 
decision-making.  Without  cross-domain  integration  and  interoperability  it  is  difficult  to  assess  the  cross¬ 
domain  impacts,  and  makes  it  difficult  to  understand  the  uncertainties,  which  is  related  to  our  sponsor's 
question  about  model  "integrity."  We  also  know  that  this  is  an  important  area  of  research  and  there  are 
various  approaches  that  are  making  progress  [7],  This  is  part  of  the  focus  on  the  path  forward. 


Designing  System 

The  Designing  System  can  be  the  entire  enterprise,  which  includes  model  capture,  synthesis,  simulation, 
analyses,  testing,  configuration  control,  workflow  automation,  product  lifecycle  management  (PLM), 
model  and  variant  management,  etc.  The  idealized  goal  is  to  transform  all  information  so  that  it  can  be 
used  for  simulation,  synthesis,  analysis  and  ideally  "models  for  manufacturing,"  (e.g.,  "3D  print  the 
system"),  models  for  training,  operations  and  sustainment.  This  is  not  realistic  for  the  entire  system  or  all 
of  the  parts,  at  least  not  today,  but  this  is  consistent  with  the  Vision  of  the  sponsor,  as  well  as  industry. 


17  bigstory.ap.org 
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We  have  had  a  number  of  discussions  with  industry  suppliers  and  users  of  technologies,  and  there  are 
technologies  that  support  multi-domain  and  multi-physics-based  ID,  2D,  and  3D  modeling,  analysis  and 
simulations. 

The  Vision  model  must  include  the  required  information  (data)  and  embedded  knowledge  that  is  normally 
captured  in  documents,  drawing,  specifications,  pseudo-formal  models,  and  tests  (some  refer  to  it  as  the 
"total"  system  model  [128]).  This  concept  was  discussed  in  terms  of  the  containing  system  [1],  designing 
system  and  ultimately  the  system  instance  that  is  the  "design."  This  includes  or  subsumes  every  piece  of 
information  that  relates  to  the  artifacts  captured  in  the  "As  Is"  process,  but  should  also  include  formalized 
information  such  as  the  inputs  and  outputs  of  modeling  and  simulations,  analyses,  surrogates,  risk 
information,  etc.  and  include  specific  versions  of  each  tool,  simulation,  and  analysis  engine  used  to 
support  the  necessary  evidence  required  to  produce  an  airworthy  system  version.  Ideally,  this  should 
include  every  piece  of  information  to  the  Bill  of  Material  (BOM). 

Discussions  with  organizations  suggest  that  some  individuals  and  organizations  understand  the  Vision 
model  concept.  Some  are  attempting  to  develop  variants  of  the  concept  that  are  more  specific  to  product 
development.  Some  have  cross-business/discipline  projects  established  to  refine  strategies  to  roll  out  and 
support  adoption  by  programs  in  these  different  business  units.  Other  efforts  are  focused  more  at  the 
software  level  (using  the  characterization  Model  Driven  Engineering  [MDE])  [64],  One  study  cited  a  multi¬ 
level,  multi-domain  instance  case  that  started  at  the  campaign  level  moving  down  through  the  mission, 
engagement,  and  engineering  levels  [3], 

There  are  also  organizations  that  claim  to  be  applying  MBSE,  yet  they  have  not  seen  the  benefits.  We 
understand  that  there  are  often  adoption  challenges  [25],  While  this  effort  was  focused  on  the  technical 
feasibility,  the  path  forward  brings  in  plans  for  organizational  adoption,  new  competencies  and  a  different 
business  operational  model  that  will  address  contracting  through  digital  information. 


Operational  Perspectives  on  a  Vision  Concept 

It  is  a  fairly  consistent  message  that  many  organizations  have  not  defined  a  Vision  model.  Instead  they 
are  involved  in  an  evolutionary  process  of  model  adoption,  and  many  want  to  better  understand  the 
return  on  investment  (ROI).  Some  organizations  have  to  address  airworthiness  and  safety-related 
requirements  and  those  efforts  can  lead  to  longer  delivery  schedules.  Even  the  automakers  are  expending 
more  resources  in  order  to  address  safety  constraints.  In  addition,  some  of  these  organizations  are 
working  on  a  subset  of  the  problem  (e.g.,  V&V)  [68],  while  others  are  approaching  this  from  the  contractor 
point-of-view,  which  is  significantly  different  from  that  of  NAVAIR.  NAVAIR  is  working  in  the  early  stages 
of  DoD  5000.02  [44]  lifecycle  (i.e.,  Milestone  A,  B,  C),  and  they  ultimately  produce  requirements  and 
design  constraints  that  are  provided  to  the  contractors.  NAVAIR's  efforts  are  focused  on  understanding 
the  problem  space  in  the  context  of  mission  analysis. 

The  objective  for  the  Vision  should  address  the  questions: 

■  Can  we  create  models  to  cover  every  type  of  artifact  that  is  required  to  produce  a  system  and 
comply  with  DoD  and  NAVAIR  processes  and  requirements  (e.g.,  Airworthiness)? 

o  In  reality,  it  is  probably  not  possible,  nor  practical;  therefore,  we  need  to  focus  on  what  is 
valuable  [94] 

■  Can  we  use  model-based  simulation,  analysis,  synthesis  and  generation  to  rapidly  traverse  the 
"Virtual  Vee"  and  continuously,  both  horizontally  and  vertically,  leverage  evolving  digital 
representations  (e.g.,  models,  surrogates)  to  assess  the  system  design  at  various  levels  of  fidelity 
in  the  context  of  continuously  evolving  mission  scenarios? 
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o  Notionally  rendered  in  Figure  3  and  Figure  23 

■  Flow  does  the  risk  framework  fit  into  the  model? 

We  initially  developed  (as  a  straw  man)  an  example  model  in  System  Modeling  Language  (SysML)  that 
represented  the  Integrated  Warfighter  Capability  (IWC),  which  broader  than  the  scope  of  the  POR.  The 
example  provided  a  common  understanding  that  the  goal  of  the  modeled  Vision  is  going  to  formally 
characterize  all  of  the  "data/'  relationships,  automation  throughout  the  entire  lifecycle,  including  for 
example  the  relationship  to  data  used  by,  and  produced  by  modeling  and  simulation,  analyses  and  other 
resources,  as  well  as  evidence  captured  within  the  models  to  support  risk  assessment  and  management.18 

We  used  SysML,  because  we  saw  examples  from  NASA/JPL  [8],  who  is  the  only  organization  that  we  met 
that  has  started  this  type  of  Vision  model  concept.  SysML  works  for  JPL,  because  their  entire  team  is 
deeply  versed  in  SysML.  However,  we  are  not  sure  about  the  approach  for  explaining  our  perspective  as 
we  also  know  that  there  may  be  many  people  in  the  NAVAIR  that  are  not  familiar  with  SysML.  Therefore, 
we  use  a  storyboard  that  was  created  with  about  10  different  views  that  should  be  more  generally 
understandable.  We  include  an  integrated  overarching  perspective  that  is  shown  in  Figure  25.  This  image 
includes  information  containment  and  operational  perspectives.  Notionally  starting  top  down  and  going 
clock-wise: 

1.  This  is  a  Collaborative  Environment 

o  We  envision  access  to  this  information  to  be  done  from  at  least  three  forms: 

•  Model  editor  form  (raw  for  the  expert  modeler,  and  this  could  include  many  types  of 
models,  DoDAF,  Simulink,  SysML,  Domain  Specific  Modeling,  Cost  model,  Computational 
Fluid  Dynamics,  Finite  Element  Models,  Reliability,  Risk,  etc.) 

•  Web-based  form;  view  of  information  synchronized  from  the  "system  model;"  we  have 
heard  many  discussion  by  tool  companies,  and  a  similar  story  about  open  MBEE  from 
NASA/JPL  [43],  and  this  is  consistent  with  technologies  discussed  by  the  commercial 
organizations 

•  This  would  allow  for  a  "dashboard"  type  web  interface,  like  the  SETR  Manager  that 
would  provide  personalized  live  updates  to  the  user;  including  prioritizing  a  user's 
workload  by  allowing  them  to  see  how  their  task  affects  the  bigger  program 

•  Documents  can  be  automatically  generated  through  personalized  or  program- 
standardized  templates 

o  Access  to  information  is  available  to  all  team  members  and  they  can  see  the  same  instance 
of  information  as  other  team  members  so  this  collaborative  environment,  which  provides  a 
single  source  of  truth;  security  mechanism  and  role-based  view  mechanisms  also  exist  today 

•  These  types  of  efforts  are  under  way  at  NAVAIR  and  more  broadly  throughout  the  Navy 
and  other  services  [117] 

2.  There  is  a  Continuous  Digital  Thread  (orange  dashed  line)  running  through  all  aspects  of  the 
concept  that  is  addressing  an  ever  evolving  set  of  needs  generically  referred  to  as  "Capability 
Sets" 

o  Continuous  digital  thread  means  that  all  digital  data  can  be  connected  and  every  piece  of 
digital  content  is  aware  of  other  digital  content;  this  is  essential  for  single  source  of  truth 
o  The  modification  of  any  item  can  trigger  events  related  to  all  other  dependencies  and  can 
change  the  state  of  that  data,  and  related  data  in  the  single  source  of  technical  truth  (e.g., 
trigger  weight  analysis  for  entire  aircraft  if  the  wing  weight  increases) 


18  There  are  number  of  useful  representations  and  documentation  that  are  not  currently  released  for  public 
viewing. 
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3.  Containing  System,  as  described  in  2.3.1,  must  represent  the  SoS,  including  environment  and 
resources  with  sufficient  fidelity  and  semantic  precision  to  understand  how  a  target  system 
interacts  within  its  environment. 

o  Capability  Sets  are  conceptually  produced  in  the  context  of  the  containing  system  through 
mission-level  modeling  and  simulation  analyses  to  address  evolving  threats/needs  as  input 
from  the  efforts  of  the  modeling  and  simulation  group 

•  We  were  provided  details  by  two  NAVAIR  groups  involved  in  mission-level  analyses,  but 
will  not  include  that  information  in  this  report  as  it  not  publically  released 

o  This  is  related  to  discussions  at  the  Mission  Level  as  reflected  conceptually  in  Figure  23  (e.g., 
operational,  and  kill  chain  scenarios,  etc.) 

4.  Program  of  Interest  should  be  an  ever  evolving  instantiation  starting  from  elements  in  the 
Reference  Model  (or  Reference  Architecture),  which  are  parts  of  the  Designing  System 

o  We  believe  that  a  model-centric  approach  to  a  radical  transformation  will  involve  the  use  of 
"Model  Measure"  or  Model  Maturity  Levels  to  assess  the  state  of  the  models' 
completeness,  well-formedness,  consistency,  etc.  and  its  ability  to  produce  all  of  the  needed 
evidence  associated  with  the  Airworthiness  constraints 
o  During  the  iterations  the  capability  sets  should  start  converging  to  a  mutually  acceptable 
program  of  interest 

o  New  technologies  and  knowledge  captured  in  the  creation  of  any  new  system  should  be 
captured  in  the  Designing  System,  including  a  continual  evolution  of  reference  architectures 
(template  of  knowledge  encapsulation  about  air  vehicle  systems  and  weapons) 

5.  Designing  System  includes  all  information  it  takes  to  go  through  analyses  and  design 
development;  this  would  include: 

o  SETR  Manager 

o  Every  modeling  and  simulation  capabilities,  ID,  2D,  3D,  SW,  HW,  System,  Mission,  etc. 
o  Trade  space  analyses 

o  Reference  model  (reference  architecture)  that  characterize  the  architectural  structures  of 
air  vehicle  systems 

•  Attributes  associated  with  data  about  those  system/subsystem/components 

•  Airworthiness  constraints 

•  Tools  that  are  used  to  provide  analyses  for  those  different  subsystems  (e.g.,  Simulink  for 
control  laws) 

o  Cost  models  linked  to  the  reference  architectural  elements 

o  Tools  such  as  Dakota  for  Quantification  of  Margins  and  Uncertainty  (QMU) 

o  Other  risk  modeling 

o  Cost  and  schedule  modeling  and  tracking 

o  System  Integration  Labs 

o  Surrogates,  hardware,  software 

o  New  tools 

o  New  approach  for  characterizing  modeling  maturity  measures 
This  list  and  story  is  not  exhaustive. 
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Figure  25.  Overarching  Concept  for  Vision 


Scope  to  Program  of  Record  through  Digital  Critical  Design  Review 

Figure  23  puts  the  scope  of  the  POR  into  context,  as  well  as  making  the  context  of  a  POR  part  of  an 
evolving  platform.  This  too  abstractly  reflects  on  the  boundaries  between  the  POR  and  the  mission  level. 
The  scope  of  this  research  task  has  been  reduced  to  focus  on  the  lifecycle  phases  up  to  critical  design 
review  (CDR),  for  the  "As  Is,"  Airworthiness,  Vision  model  and  risk  framework  for  a  POR.  It  was  thought 
that  the  technical  reviews  are  good  "checkpoints"  since  they  focus  on  different  decisions  and  levels  of 
engineering  content  that  would  need  to  be  represented  in  the  models.  Only  the  PDR  and  CDR  are  always 
required.  Other  reviews  such  as:  ASR,  SRR,  SFR,  TRR,  SVR,  PRR  may  or  may  not  be  required  on  a  given 
program.  Ideally,  we  are  looking  for  a  new  concept:  Digital  design  from  CDR  artifacts  (DCDR).  We  want  to 
investigate  a  more  continuous  notion  of  PDR  and  CDR  (or  DCRD)  where  reviews  of  models  and  analysis 
artifacts  occur  "everyday"  until  all  of  the  required  evidence  is  provided  in  order  to  support  contracts  and 
signoffs;  any  meeting  can  be  virtual  and  in  real-time  when  data  is  available.  This  concept  is  now  part  of 
the  new  SETR  Manager. 

More  importantly,  now  that  we  have  evidence  about  some  aspects  of  the  technical  feasibility  question, 
we  want  to  understand  if  there  are  alternative  types  of  model  measure  that  can  be  used  to  supplement 
or  eliminate  these  traditional  document-centric  reviews  as  part  of  the  radical  transformation.  Part  of  the 
ongoing  research  is  to  investigate  if  such  a  concept  is  viable. 
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Context  for  Program  of  Record  System 


The  context  for  the  POR  starts  from  environmental  aspects  at  the  mission-level.  For  many  efforts 
organizations  often  start  with  a  DoDAF  operational  view  (OV-1)  diagram  of  the  mission-level  with 
systems-of-system  (SoS)  level  interactions;  increasingly  many  are  using  dynamic  OVls  such  as  those 
reflected  in  Figure  5,  which  aligns  better  with  the  model-centric  engineering  concept.  The  operational 
views  decompose  the  mission  within  the  context  of  the  situation,  and  provide  different  viewpoints  that 
describe  the  tasks  and  activities  operational  elements,  and  resource  flow  exchanges  required  to  conduct 
operations  related  to  scenarios,  as  reflected  in  Figure  24. 

NAVAIR  Mission  Level  Modeling  and  Simulation  (M&S) 

We  had  a  discussion  with  two  NAVAIR  M&S  groups  who  are  responsible  for  analyzing  the  mission 
scenarios.  The  groups  do  have  a  vision,  but  not  a  model  of  that  vision.  They  indicated  that  there  will  be 
much  more  cross-domain  integration,  but  the  current  capabilities  appear  not  to  have  much  integration. 
The  views  from  these  M&S  capabilities  (i.e.  capability  sets  in  Figure  25)  define  what  we  have  discussed  as 
the  containing  system  part  of  the  Vision  model,  but  currently  they  are  not  integrated.  For  our  research 
task  scoped  at  the  POR,  this  information  is  on  the  interface  boundary,  but  there  is  not  much  that  feeds 
down  today;  that  is,  the  majority  of  the  analyses  from  the  M&S  groups  are  focused  upwards  towards  the 
campaign  level,  rather  than  downwards  towards  the  system  (aka  engagement  level). 

Model-centric  perspectives  at  the  POR  level  would  be  potentially  useful  for  this  effort,  because  their  M&S 
capabilities  must  often  create  some  type  of  abstraction  of  the  PORs  and  platforms.  This  is  part  of  the  plan 
for  the  future,  which  is  to  better  understand  the  interface  boundary  between  the  M&S  level  and  the  POR 
level  within  the  context  of  the  Vision. 

NAVAIR  Study  Views 

Study  views  were  created  to  address  a  number  of  challenges  at  the  POR  level  and  in  creating  DoDAF 
requirements.  The  study  view  concept  builds  on  lessons  learned  from  creating  early  DoDAF  models; 
analyses  have  uncovered  that  interoperating  at  the  lowest  (data)  levels  is  insufficient  for  scenarios,  and 
scenarios  require  behaviors,  which  is  missing  at  the  data  level.  DoDAF  does  not  accommodate  other 
scenario  requirements  (e.g.,  conditions,  assumptions)  very  well,  and  is  insufficient  to  fully  characterize 
the  dynamics  needed  for  analysis. 

A  mission-level  SoS  analysis  begins  with  formalization  using  Study  Views,  as  reflected  in  Figure  26,  which 
has  M&S  dynamic  views  and  visualization.  Study  views  provide  structure  and  a  common  context  that  acts 
as  a  basis  for  framing  and  bounding  the  functional  decomposition  of  DoDAF  products.  Study  views 
formalize  the  need  and  intent,  provide  a  situational  context  and  influencing  factors  to  frame  and  bound 
the  functions  and  activities  of  the  mission  and  scenarios  that  ultimately  lead  into  corresponding 
representations  of  the  Mission  and  System  Capabilities  (i.e.,  the  capabilities  for  the  POR).  These  capability 
representations  are  further  analyzed  using  modeling  and  simulation  and  corresponding  analysis 
capabilities.  The  outputs  of  which  are  then  formalized  in  terms  of  DoDAF  artifacts  by  the  NAVAIR 
Architecture  group.  This  information  forms  the  analysis  boundaries  for  the  System  Capabilities 
information  needed  to  define  requirements  for  the  POR. 
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Figure  26.  Mission  Context  for  System  Capability19 


Structuring  Mission-Level  Analysis 

NASA/JPL,  like  NAVAIR,  has  created  their  own  supporting  tool  that  provides  for  the  structured  entry  and 
retrieval  of  architecture  artifacts  based  on  an  emerging  architecture  metamodel.  We  summarize  some 
aspects  of  it  here,  because  it  goes  beyond  what  we  currently  know  about  Study  Views  and  is  being  applied 
and  evolved  on  Jupiter  Europa  Orbiter  (JEO)  project  [100], 

The  architecting  focus  was  elevated  to  a  more  prominent  and  formal  role  on  the  JEO  project  than  has 
been  done  on  most  other  NASA/JPL  projects;  the  emphasis  is  to  make  systems  engineering's  basic 
processes,  such  as:  requirements  generation,  trade  studies,  risk  management,  design  and  interface 
control,  verification  and  validation,  etc.,  more  coherent.  The  new  architecting  process  used  on  the  JEO 
project  and  framework  is  intended  to  aid  systems  engineering  in  the  following  ways: 

■  Adding  guiding  structure 

■  Providing  better  integration  of  the  resulting  artifacts 

■  Ensuring  comprehensive  attention  to  important  relationships 

■  Facilitating  broad  understanding  of  the  architecture 

■  Maintaining  system  integrity  over  the  course  of  development 


19  Image  source:  Thomas  Thompson,  Enabling  Architecture  Interoperability  Initiative,  B210-001D-0051 
Unclassified. 
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■  Helping  to  ensure  comprehensive  verification  and  validation  (V&V) 

NASA/JPL  acknowledged  the  choice  of  a  different  framework  (e.g.,  not  DoDAF,  which  is  used  by  NAVAIR), 
because  they  viewed  the  choice  of  framework  should  be  dependent  on  the  nature  of  the  system  and 
circumstances  it  was  designed  to  support.  The  JEO  most  closely  aligns  with  the  emerging  ANSI/IEEE  1471- 
2000  standard  [67]  for  software-intensive  systems.  The  architecture  artifacts  include,  but  are  not  limited 
to,  Stakeholders,  Concerns,  Viewpoints,  Views,  Analyses,  Models,  Elements,  Scenarios,  Properties,  and 
Functions,  which  align  with  many  of  the  Study  View  concerns. 

The  JEO  project  team  efforts  have  focused  on  five  objectives: 

■  Identifying  and  capturing  stakeholders  and  their  concerns 

■  Developing  the  content  for  and  capturing  viewpoints  and  views  related  to  the  concerns 

■  Identifying  and  initiating  trades  that  are  needed  in  the  near-term 

■  Maturing  the  models  that  are  needed  to  support  those  trades 

■  Training  for  the  growing  architecting  team 

The  JEO  project  team  are  leveraging  the  concept  and  tooling  to  support  a  single  source  of  truth20  [35], 
using  modeling  patterns  to  embed  methodological  guidance,  and  computationally  applying  formal 
reasoning  to  ensure  that  the  models  are  consistent  and  satisfiable  [69],  with  respect  to  the  ontologies  for 
these  various  patterns. 

Reference  Architecture  &  Model  Based  Engineering  Environment 

The  NASA/JPL  projects  have  a  related  reference  architecture  and  associated  open  Model  Based 
Engineering  Environment  (Open-MBEE)  [43]  that  they  are  using  and  evolving  on  the  JEO  project.  The 
reference  architecture  aligns  with  the  vision  model  concept.  They  used  MagicDraw,  which  supports 
SysML/UML  [92]  and  other  modeling  capabilities  to  define  activities  that  are  transformed  to  an  Oracle 
database  to  manage  workflow.  MagicDraw  also  provides  support  to  plug-in  domain  specific  modeling 
tools  [79],  They  are  modeling  their  artifacts  and  activities  to  generate  the  controls  for  a  workflow  engine. 

Figure  27  provides  an  overarching  perspective  on  one  of  the  views  extracted  from  a  report  [9]  that  is 
applicable  to  the  Vision  model: 

■  Blocks  in  the  diagram  define  categories  of  items  requiring  exposition  in  the  architecture 
description 

o  Accompanying  each  category  is  a  template  (not  shown)  specifying  the  sorts  of  information 
required  for  each  member  of  that  category 

■  Stakeholders  and  their  Concerns  are  the  drivers  for  everything  else  in  the  architecture,  i.e.,  they 
can  be  considered  the  'entrance  points'  to  explore  the  framework 

o  This  is  somewhat  analogous  to  the  purpose  of  the  Study  Views  developed  at  NAVAIR, 
although  NAVAIR  does  not  have  a  similar  representation  of  its  context  in  a  model 
representation 

■  The  Element  is  a  place  holder  for  aspects  of  the  System  to  be  designed  (i.e.,  Program  of  Interest 
in  Figure  25) 

■  The  Models,  the  Analyses  performed  on  them,  and  the  Scenarios,  which  relate  to  the 
"Containing  System"  (e.g.,  for  a  Program  of  Record)  complete  the  blocks  of  the  Architecture 
Description 


20  NASA  uses  the  term  single  source  of  truth,  while  NAVAIR  has  adopted  the  term  single  source  of  technical  truth. 
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Figure  27.  NASA/JPL  Architecture  Framework  Tool  (AFT)  for  Architecture  Description 

This  concept,  with  the  information  provided  on  the  reference  architecture  and  the  associated  modeling 
patterns  provides  one  of  the  best  stories  we  have  heard  as  it  relates  to  formalizing  the  concept  of  the 
Vision  model.  This  perspective  informed  our  development  of  a  NAVAIR-oriented  concept  reflected  in 
Figure  25. 

NAVAIR  Architecture  Group 

The  inputs  from  the  M&S  group,  such  as  Study  Views  as  shown  in  Figure  26  are  inputs  to  the  System 
Requirements  Analysis  and  Architecture,  which  focuses  on  developing  DoDAF  views  to  drive  the  system 
analysis  and  design.  They  are  working  toward  the  requirements  for  the  l°Mlaval  Enterprise  Architecture 
Repository  (NEAR),  which  includes  the  need  for  Physical  Exchange  Specification  (PES)  compliance, 
however  this  is  a  challenge,  because  some  of  the  tools  do  not  support  PES  in  the  same  way.  While  these 
efforts  are  using  models,  they  are  not  using  dynamic  models.  Most  important  is  that  these  DoDAF  type 
models  are  increasingly  being  used  in  communications  with  system  contractors.  While  this  is  not 
necessarily  a  radical  transformation,  it  continues  to  support  the  concept  that  sharing  information  through 
models  is  happening  today. 
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Modeling  and  Tools  for  the  Vision 


Our  team  has  had  numerous  discussions  about  modeling  representations,  languages  and  tools  for  the 
Vision.  We  believe  there  will  be  many  languages  and  modeling  representations  that  address  both 
traditional  system  engineering  views  such  as  SysML,  but  also  physics-based  and  other  modeling  views. 
There  will  also  be  increasingly  much  more  dynamic  information.  We  will  also  see  an  increasing  use  of 
Domain  Specific  Models  and  associated  languages  [24]  [79]  [94],  and  leverage  transformations  to  support 
analysis,  generation,  and  alternative  types  of  visualization. 

The  basic  SysML  diagrams  in  the  modeling  environments  are  mostly  static.  System  engineering  models 
defined  in  SysML  are  descriptive  in  nature  and  do  not  directly  produce  analytical  results  [68],  nor  are  they 
executable  [32],  Different  tool  vendors  provide  extensions  or  their  own  analytical  capabilities  that  solve 
SysML  parametric  diagram  [26],  Since  the  parametric  relationships  are  solved  as  a  system  of  equations, 
the  analytical  model  is  limited  to  simple  equations.  To  be  able  to  use  more  sophisticated  engineering 
analyses,  external  analysis  tools  need  to  be  connected  to  SysML  models.  Other  research  efforts  are 
attempting  to  leverage  other  standard  modeling  languages  such  as  Modelica  [92]  (see  Figure  Figure  8) 
that  have  a  broad  range  of  analytical  support  through  integration  between  SysML  and  Modelica. 

Modelica  is  a  standardized  general-purpose  systems  modeling  language  for  analyzing  the  continuous  and 
discrete  time  dynamics  of  complex  systems  in  terms  of  differential  algebraic  equations.  Domain  Specific 
Modeling  environments  (e.g.,  Simulink  for  control  systems)  often  have  richer  semantics  (e.g.,  structure, 
behavioral  and  sometimes  temporal)  to  support  dynamic  analyses  and  simulation;  some  also  have  formal 
method  analysis  and  automated  test  generation  [13]  [22]  [99],  Other  approaches  provide  process 
integration  and  design  optimization  framework  allowing  for  many  available  analysis  solvers  or  custom 
solvers  for  all  type  of  analysis  with  simulation  and  workflow  automation  [73], 

Because  SysML  is  general,  there  are  possible  mappings  to  many  types  of  modeling  languages  (as  is  true 
for  UML  also)  [129]  as  well  as  support  for  programmatic  interchange  based  on  the  XML  Metadata 
Interchange  (XMI)  standard  [91].  This  may  rationalize  why  some  organizations  are  using  SysML  as  an 
integrating  framework,  that  is,  they  may  not  be  modeling  in  SysML,  but  they  are  using  SysML  (and 
associated  tooling)  as  a  mapping  or  an  interchange  medium  between  different  modeling  languages  and 
environments  [8]  [43],  While  the  SysML  and  UML  languages  and  tools  help  significantly  to  formalize  the 
expression,  exchange,  and  graphical  representation  of  system  models,  SysML  and  UML  languages  remain 
ambiguous  and  in  need  of  extensions  to  capture  the  specific  semantics  of  a  given  engineering  domain 
[117]. 

The  perspectives  cited  in  this  section  reflect  on  why  the  Vision  must  go  beyond  and  use  other  more 
semantically  rich  and  precise  model  representations,  as  well  as  supporting  semantically  consistent  model 
(digital)  interchange  between  different  simulation  and  analysis  tools.  Our  efforts  planned  for  the  next 
phase  will  investigate  a  potentially  more  general  approach  for  representing  the  Vision,  and  a  tool  agnostic 
approach  to  the  single  source  of  technical  truth,  which  we  think  can  support  the  entire  lifecycle  [7]  [124], 


Model-Centric  Engineering  Environment  Perspectives 

This  section  provides  additional  information  related  to  discussions  or  actions  from  our  working  sessions 
to  reflect  on  a  scenario  for  how  MCE  tool  chains  can  provide  semantically  formalized  information  between 
tools  to  subsume  what  would  normally  be  manual  processes. 
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Model  Transformation  Rather  Than  Model  Evolution 


To  reflect  on  the  concept  of  model  transformation  rather  than  model  evolution,  we  provide  the  following 
example  to  describe  how  model-based  automation  can  completely  eliminate  manual  effort  and  result  in 
radical  transformation  of  the  "As  Is"  process  through  an  automated  workflow  [18].  This  process  was  used 
in  the  verification  of  the  control  laws  for  the  F-35  [88],  This  scenario  relates  to  a  NAVAIR  "As  Is"  artifact 
called  the  "Flight  Control  Detailed  Design  Report."  In  a  model-centric  world  this  type  of  artifact  would: 

■  Represent  "Control  Law"  in  a  model 

o  Simulink21  and  Stateflow  are  commonly  used  to  model  control  laws  (e.g.,  F-16,  F-18,  F-35) 

[88] 

■  Automated  analysis  that  exists  today,  (e.g.,  it  has  been  applied  to  F-35)  would  include: 
o  Satisfiability:  proving  that  each  thread  through  the  model  has  no  contradictions 

(mathematical  consistency) 

■  Simulation 

o  Simulation  of  Simulink  models  is  often  done  using  Matlab 
o  Support  high-fidelity  simulation  using  Matlab 

o  Support  higher  fidelity  with  real-time  execution  within  the  surrogate  or  prototype  system 
implementation  or  actual  hardware  though  automatic  code  generation 

■  Synthesis  or  generation 

o  Code  generation  from  Simulink  models  can  be  provided  by  Mathworks  and  other 
commercial  products 

o  Automatic  test  generation  directly  from  Simulink  models  [18] 
o  Automatic  test  driver  generation 

■  The  test  vectors  are  transformed  into  various  types  of  test  drivers  that  are  run  both  against  a 
Matlab  simulation  and  the  auto-generated  code;  if  all  tests  pass  (the  actual  output  equals  the 
expected  output)  in  both  the  simulation  and  generated  code  execution  environments  then  there 
is  a  strong  verification  argument  that  the  code  satisfies  the  specification 

o  Organizations  run  the  test  through  both  the  simulation  and  code,  because  organizations 
have  been  able  to  find  errors  in  the  code  generation  (Risk  reduction  argument  for  using 
model-based  tools) 

■  Code  coverage  tools  such  as  LDRA  and  VectorCAST  have  been  used  to  show  that  the  tests 
provide  Modified  Condition/Decision  (MC/DC)  coverage  [59] 

o  Code  coverage  measurement,  which  provides  quantified  risk  reduction  evidence 

■  The  Mathworks  code  generation  uses  a  particular  algorithm  that  produces  code  that  is 
"deadlock"  free 

o  Eliminates  concurrency  analysis 

These  are  types  of  model-based  automation  that  leverage  models  to  "Cross  the  Virtual  V."  While  this  can 
be  and  is  commonly  done  on  low-level  high-fidelity  models,  we  are  also  interested  in  applying  this  type 
of  concept  at  the  upper-levels  of  the  "V"  with  varying  levels  of  fidelity  that  provide  integration  of  model 
and  model  automation  at  different  levels  of  the  "V." 

This  is  a  positive  story  as  it  relates  to  the  use  Simulink-based  modeling  tool  chains  that  can  significantly 
reduce  time  by  both  supporting  simulation,  code  generation,  analysis  and  test  generation.  However, 
other  forms  of  software  modeling  have  not  had  this  same  type  of  automation,  because  behavioral 


21  We  are  not  promoting  Simulink,  we  use  it  as  an  example,  because  it  is  almost  a  defacto  standard  for  control 
system  modeling  and  simulation,  and  it  was  the  tool  used  in  the  above  scenario. 
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information  in  a  modeling  framework  ( e.g UML  Rhapsody)  is  manually  coded,  and  that  cannot  be 
analyzed  in  the  same  way  that  Simulink  models.  This  is  a  concern  as  software  is  growing  in  complexity 
and  size. 


Crossing  the  Virtual  "V"  by  Leveraging  Models,  Digital  and  Physical  Surrogates 

We  have  continually  discussed  the  notion  of  "Crossing  the  Virtual  V"  as  an  important  way  to  assess  system 
design  concepts  in  the  context  of  mission  scenarios.  However,  in  discussions  with  organizations,  there  are 
some  that  believe  that  the  notion  of  a  "V"  is  a  historic  manifestation  of  "the"  traditional  flow  of  document- 
driven  work,  and  we  should  eliminate  the  use  of  the  "V"  as  part  of  systems  engineering  dogma  as  it  is 
counterproductive  to  embracing  approaches  that  support  the  continuous  integration  of  digital  artifacts. 
The  "V"  introduces  points  for  disconnects  and  failure.  What  is  more  critical  in  the  Vision  is  continuous 
integration  of  various  types  of  digital  assets  with  varying  levels  of  abstraction  and  various  degrees  of 
fidelity  as  reflected  in  Figure  3.  Some  discussion  on  this  point  using  examples  to  further  clarify  the  notion 
of  physical  surrogates,  and  support  the  argument  that  the  "V"  may  not  be  a  good  metaphor. 

The  concept  of  MCE  relies  heavily  on  digital  assets  such  as  physical  surrogates,  existing  system,  or 
component  re-purposed  for  new  concept  exploration  and  prototyping.  Our  NAVAIR  team  created  a 
concept  for  representing  System  Level  Maturity.  It  reflects  on  the  idea  that  as  we  are  attempting  to  "Cross 
the  Virtual  V"  and  will  rely  on  physical  surrogates,  which  is  commonly  done  today,  both  in  aerospace  and 
other  domains,  such  as  auto  racing.  The  actual  airframe,  shown  Figure  3  along  the  bottom  matures  (right- 
to-left)  and  the  actual  aircraft  is  first  flow  (e.g.,  F-35,  15-December-2006)  long  before  many  of  the 
complex  software  intensive  systems  are  developed  and  integrated,  as  the  aircraft  airframe  and  new 
materials  are  being  evaluated.  Key  early  capabilities  such  as  software  for  the  control  laws  to  fly  the  aircraft 
are  often  evolved  from  earlier  aircraft  systems  (e.g.,  many  versions  of  MATRIXx  and/or  Simulink  models 
have  been  evolved  for  years,  and  will  continue  to  be  evolved  for  years).  Yet,  all  of  these  systems  are 
continually  refined  and  as  the  timeline  of  system  capabilities  mature,  new  capabilities  are  added  to  the 
system.  We  believe  that  in  MCE,  it  will  be  possible  to  have  continuous  integration  and  tests,  much  like 
agile  is  used  in  software.  Formalized  interfaces  are  required  for  integration,  and  the  semantics  for  the 
interfaces  often  need  to  be  formalized:  1)  structurally,  2)  behaviorally,  and  3)  temporally,  in  order  to  use 
surrogates  and  simulations.  Document-based  specifications  do  not  formalize  these,  however  some 
modeling  approaches  can,  and  with  semantic  formalization,  automated  verification  can  be  supported 
directly  from  the  models,  which  is  supported  by  evidence. 


Decision  Framework 

The  Army's  TARDEC  provided  a  presentation  and  demonstration  on  an  evolving  a  framework  called  the 
Integrated  System  Engineering  Framework  (ISEF)  [52]  [117].  Briefly,  ISEF  is  a  Web-enabled  collaborative 
decision-making  framework  to  support  knowledge  creation  and  capture  built  on  a  decision-centric 
method,  with  data  visualizations,  that  enables  continuous  data  traceability.  The  framework  integrates  a 
number  of  different  technologies  that  support  decision-making  applicable  to  different  phases  of  the 
lifecycle,  for  example: 

■  Requirements  -  they  have  their  own  requirement  management  capability 

■  Feedback  mechanism 

■  Portfolio  management 

■  Architecture  (through  other  MBSE  tools) 

■  Tradespace  analysis 

■  Risk 
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Road  mapping 


While  the  information  from  this  meeting  may  not  directly  address  the  research  question  for  a  radical 
transformation,  the  information  we  received  seems  valuable,  as  ISEF  is  complementary  for  a  transitional 
model-centric  approach.  It  also  illustrates  the  need  for  organizations  to  do  some  level  of  architecting  even 
if  they  integrate  other  tools.  This  is  likely  to  be  the  case  for  NAVAIR,  because  they  will  need  to  have  a 
more  tool  agnostic  environment. 


Systems  Engineering  Technical  Review  (SETR)  Manager 

This  section  briefly  discusses  the  new  SETR  Manager,  which  is  inherently  part  of  the  "As  Is"  process,  but 
we  believe  it  could  be  part  of  the  "to  be"  Vision.  Similar  in  some  respects  to  ISEF,  the  SETR  Manager  is  a 
server/web-enabled  way  to  navigate  through  the  SETR  checklists.  It  provides  real-time  status  updates 
and  reviews,  and  allows  for  discussion  tracking  providing  a  familiar  Facebookand  Twitter  style  that  should 
provide  an  easy-to-use  look  and  feel,  allowing  teams  to  come  up  to  speed  quickly.  This  capability  is  a 
transformation  of  a  few  different  types  of  SETR  checklist  approaches  that  have  structured  and  layered 
different  types  of  tooling  for  the  checklist  with  some  reorganization  of  the  checklist  questions  (more 
5000),  but  layering  them.  The  Tier  4  questions  (~1500)  are  still  Yes/No,  and  the  other  possible  question 
have  now  been  moved  to  Tier  5,  and  are  referred  to  as  Considerations,  which  add  context  to  the  Tier  4 
questions.  There  may  be  a  need  to  move  some  of  the  Tier  5  questions  to  Tier  4. 

SETR  Manager  will  be  an  ongoing  evolution,  which  NAVAIR  wants  to  do  in  a  much  more  iterative  (Agile¬ 
way).  In  its  current  state  the  SETR  Manager: 

■  Provides  dashboard  views  of  the  SETR  Manager  data  for  all  primary  management  roles,  and 
competency  (tech  authority,  SETR  content  owner) 

■  Uses  the  dashboard  to  support  drill-down  of  data 

■  Visualizes  historical  trends  where  possible 

■  Allows  comparisons  between  different  sets  of  data  (i.e.  between  multiple  competencies  or 
programs) 

■  Steers  attention  quickly  toward  potential  issues  and/or  tasks  that  must  be  accomplished 

The  SETR  Manager  is  part  of  the  Designing  System,  shown  in  Figure  25,  and  also  believe  it  to  be  a  key 
element  of  the  "rich"  web  interface  into  the  single  source  of  technical  truth.  The  overall  metaphor 
provided  by  the  capability  aligns  with  a  much  more  collaborative  way  of  supporting  real-time  reviews  and 
consolidated  measurements  in  consistent  colorized  dashboards,  with  visualization.  The  server-based 
approach  allows  for  an  easier  and  more  continuous  updates  as  NAVAIR  adapts,  and  to  support  integration 
of  other  web-enable  and  server-based  approaches  for  continuous  and  collaborative  engineering. 


Integrated  MCE  Environment 

We  have  provided  a  number  of  references  and  showed  images  representing  overarching  MCE  concepts 
that  have  been  created  by  other  organizations,  such  as  Figure  12.  There  are  other  notional  perspectives 
created  by  many  organizations,  and  we  use  Figure  28  as  a  notional  summary  provided  in  yet  another 
perspective  of  an  integrated  environment  of  capabilities  for  the  vision  concept: 

■  Provides  appropriate  views  for  the  various  stakeholder 

■  Stakeholders  have  views  into  the  Single  Source  of  Technical  Truth  (SSTT) 

■  Using  rich  modeling  interfaces  for  those  with  expertise  in  modeling 


61 


■  Using  rich  "web"  interface,  which  today  provides  support  for  graphics,  integrated  with 
structure  inputs,  generated  textual  views  and  3D  model  viewing  [102] 

■  MDAO  layer  provides  for  problem  and  design  space  exploration  of 
o  Physics-based  models 

o  Integrity-based  models 
o  Cost  and  scheduling  models 
o  Risk  models 
o  Various  "illities"  models 
o  Including  surrogates  and  components 

■  Enabled  by  High  Performance  Computing  (HPC) 

■  Semantically  rich  linkages  between  data  and  information  in  the  SSTT  provides  for  continuous 
workflow  orchestration  -  enabled  by  HPC 

■  Document  generation  is  enabled  by 

o  Semantically  rich  links  to  information  in  the  SSTT 
o  Templates  that  formalize  patterns  for  requirements,  contracts,  etc. 

■  Enabling  technologies  such  as  machine  learning  provides  a  virtual  knowledge  librarian  that  assist 
users  guided  by  embedding  knowledge  and  training 

■  Contractor  and  collaborators  have  a  secure  means  to  plugin  to  view  or  share  digital  information 
as  a  new  paradigm  for  interactions 

■  This  view  of  the  Designing  System  provides  links  downstream  to  fully  link  PLM 
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Figure  28.  Integrated  Environment  for  Iterative  Tradespace  Analysis  of  Problem  and  Design  Space 
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Task  4  -  Integrated  Framework  for  Risk  Identification  and  Management 


We  are  researching  strategies,  methods,  and  tools  for  a  risk-based  framework  that  aligns  with  the  Vision 
model  concept  through  MCE.  This  involves  how  the  Vision  model  should  include  integrated  risk 
identification  and  management.  While  there  are  many  classes  of  risks  to  manage,  for  NAVAIR  there  are 
fundamentally  two  key  classes  of  risk  that  we  have  been  asked  to  consider: 

■  Airworthiness  and  Safety  (most  critical  in  Technical  Feasibility  assessment) 

■  Program  execution  (cost,  schedule  and  performance) 

There  are  also  two  complementary  views  of  model-based  acquisition  with  respect  to  risk: 

■  Risks  introduced  by  modeling  deficiencies  and  risks  reduced  by  enhanced  use  of  modeling 

■  Modeling  to  predict  or  assess  risks  (i.e.,  modeling  for  uncertainty  quantification  in  acquisition 
and  in  the  use  of  models) 

The  risk  framework  must  also  addresses  the  sponsor's  question: 

If  we  are  going  to  rely  more  heavily  on  model-centric  engineering,  with  an  increasing  use  of  modeling 
and  simulations,  how  do  we  know  that  models/simulations  used  to  assess  "performance"  have  the 
needed  "integrity"  to  ensure  that  the  performance  predictions  are  accurate  (i.e.,  that  we  can  trust 
the  models)? 

This  brings  in  the  need  for  approaches  to  what  has  been  traditionally  referred  to  as  Verification, 
Validation  and  Accreditation  (VV&A)  of  modeling  and  simulation  capabilities.  VV&A,  in  principle,  is  a 
process  for  reducing  risk;  in  that  sense  VV&A  provides  a  way  for  establishing  whether  a  particular 
modeling  and  simulation  and  its  input  data  are  suitable  and  credible  for  a  particular  use  [47],  The  word 
tool  qualification  [48]  and  simulation  qualification  [106]  have  also  been  used  by  organizations  regarding 
the  trust  in  models  and  simulations  capabilities. 

There  is  also  a  concern  that  the  risk  of  SE  transformation  to  MCE  will  fail  to  provide  an  efficient,  effective 
and  reliable  alternative  to  the  current  process.  This  is  an  important  subject,  but  not  address  in  this  section. 

This  sections  covers  many  elements  needed  for  a  risk  framework,  but  is  not  exhaustive,  such  as: 

■  Risk  context  for  the  other  topics  covered  in  this  section 

■  Risk  consequences  from  model  centric  engineering 

■  Scope  of  the  risk  framework,  which  is  fundamentally  based  on  transitioning  from  document¬ 
centric  processes  to  model  centric  engineering  in  assessing  risk 

■  Model  validation  and  simulation  qualification 

■  Modeling,  methods  and  tools  for  quantification  of  margins  and  uncertainty 

■  Risk-informed  predictive  models  for  risk  identification  based  on  subjective  information 

■  Examples  for  improving  the  integrity  of  model 

■  Model-centric  methods  and  tools  enabling  approaches  to  safety  and  airworthiness 

■  Risk  in  a  radically  transformed  and  collaborative  environment 

■  Risk-related  research 


Risk  Context 

Defined  in  the  DoD  Risk  Management  Guide  [46], 

Risk  is  a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and  objectives 
within  defined  cost,  schedule  and  performance  constraints.  Risk  can  be  associated  with  all  aspects  of 
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a  program  (e.g.,  threat,  technology  maturity,  supplier  capability,  design  maturation,  performance 
against  plan).  Risk  addresses  the  potential  variation  in  the  planned  approach  and  its  expected 
outcome. 

Risks  have  three  components: 

■  A  future  root  cause  (yet  to  happen),  which,  if  eliminated  or  corrected,  would  prevent  a  potential 
consequence  from  occurring, 

■  A  probability  (or  likelihood)  assessed  at  the  present  time  of  that  future  root  cause  occurring,  and 

■  The  consequence  (or  effect)  of  that  future  occurrence. 

A  future  root  cause  is  the  most  basic  reason  for  the  presence  of  a  risk.  Accordingly,  risks  should  be  tied 
to  future  root  causes  and  their  effects.  A  risk  framework  needs  to  address  how  MCE  can  identify  future 
risk  and  characterize  its  margins  and  uncertainty  in  the  face  of  continual  change  of  the  problem  analysis 
and  design. 


Risk  of  Consequence  from  Model  Centric  Engineering  Transformation 

A  concern  is  risk  of  adverse  consequences  resulting  from  radical  transformation  to  MCE  acquisition. 
Possible  adverse  consequences  of  concern  are  (a)  failure  to  produce  aircraft  that  can  be  certified  as  safe 
and  airworthy,  (b)  failure  to  be  able  to  certify  airworthiness  and  safety,  and  (c)  certifying  unsafe  or 
unworthy  systems  as  safe  and  airworthy.  We  are  not  addressing  the  risk  that  MCE  transformation  fails  to 
produce  the  desired  reduction  in  acquisition  time  and  cost. 

We  assume  that  radical  transformation  to  MCE  acquisition  will  not  involve  radical  change  to  the 
airworthiness  certification  criteria  (e.g.,  MIL-HDBK-516B/C  [80]),  or  system  safety  goals,  objectives  and 
analysis  framework.  However,  we  do  believe  that  the  production  of  the  evidence  needed  will  be  done  in 
a  very  different  way  derived  primarily  from  models  and  the  associated  analytical  means. 

We  assume  that  transformation  to  MCE  will  have  several  major  effects  on  the  airworthiness  and  safety 
certification  process.  We  assume  that  manual  reviews  and  analyses  of  paper-based  requirements,  design, 
engineering  and  manufacturing  documentation  will  be  replaced  with  analysis  of  executable  models  and 
analysis  using  executable/dynamic  models  (i.e.,  analysis  of  the  models,  and  analysis  with  the  models)  with 
interactive  visualizations  [41].  We  assume  that  test  design  and  analysis,  at  all  levels  of  the  system,  will  be 
conducted  in  an  iterative  process  in  which  models  will  be  used  to  define  the  conditions  for  the  next  test 
(experimental  design)  and  to  analyze  the  test  results.  We  assume  that  models  of  the  system,  models  of 
the  test  process  and  instrumentation,  and  models  of  the  uncertainty  in  the  system  models  will  be  used 
to  define  tests  that  will  produce  the  greatest  possible  reduction  in  (a)  uncertainty  regarding  airworthiness 
and  safety,  and  (b)  reduction  in  uncertainty  with  regard  to  the  validity  of  the  models  and  the  inputs  to 
the  models.  We  assume  that  results  of  the  testing  will  be  used  to  refine  the  models  and  their  calibration 
data,  as  well  as  being  used  to  score  the  system  with  respect  to  airworthiness  and  safety  certification 
criteria. 

The  risk  assessment  framework  consists  of  identifying  the  major  areas  or  types  of  risks  resulting  from 
transformation  to  MCE,  and  assessing  whether  those  risks  are  manageable  (i.e.,  the  feasibility  of  effective 
risk  management).  Risk  management  consists  of  identifying  risks,  quantifying,  planning  and 
implementing,  detecting,  mitigating,  and  monitoring  detection  and  mitigation,  and  estimating 
uncertainties  in  them. 

Some  major  risk  types  and  areas  that  we  identified  are: 
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■  Models  do  not  have  adequate  resolution,  completeness  and  fidelity  to  be  used  to  address  the 
airworthiness  and  safety  criteria 

■  Models  do  not  have  adequate  fidelity  with  respect  to  the  manufacturing  process,  manufactured 
test  articles  and  test  implementation 

■  Models  of  human  behavior  and  performance  are  inadequate  with  respect  the  range  of  human 
errors  and  processing  limitations,  and  simulators  of  man-in-the-loop  testing  fail  to  adequately 
simulate  the  phenomena  in  the  operating  environment 

■  Adaptive  nature  of  the  iterative  model-test-model  cycle  leads  to  homing  in  on  specific  areas  of 
uncertainty  while  avoiding/ignoring  others 

■  Unstated  assumptions  in  the  airworthiness  criteria  and  in  the  models  are  inconsistent  and 
incompatible 

■  Process  of  model  calibration  and  validation  with  respect  to  airworthiness  and  safety  concerns 
requires  the  same  procedures,  tests,  reviews  and  analyses  as  the  current  airworthiness  and 
safety  process  to  achieve  the  same  level  of  certainty 

■  Model-centric  airworthiness  and  safety  certification  will  require  more  effort  and  a  different  skill 
set  than  the  current  process 

■  Model-centric  approach  will  conceal  "blind  spots"  -  factors  and  effects  not  included  in  the 
models  will  be  ignored  or  concealed  in  certification,  test  design  and  analysis 

■  Calibration  and  validation  strategies  for  highly  non-linear  events  and  limited  test  &  observation 
opportunities 

■  Models  used  out  of  context,  outside  validation  &  calibration 

■  Limitations,  assumptions,  and  phenomena  omitted,  not  often  not  well  articulated 

■  Deterministic  chaos  phenomena  where  small  change  in  boundary  conditions  (inputs)  produces 
rapid  divergence  in  outputs  not  reflected  in  models  or  simulation  scenarios  (non-linearities) 

■  Sensitivity  to  complex  and  often  unknown  boundary  conditions 

■  Gaps  in  understanding  multi-scale,  multi-physics  phenomena,  potentially  due  to  limitations  in 
cross-domain  model  integration 

■  Human  behavior,  knowledge,  cognition  -  flight  safety,  damage  control 

■  Level  of  modeling  and  simulation  different  from  level  of  analysis  and  decision 

■  Incompatible  scope,  resolution,  terminology  with  test  procedures 

The  standard  for  acceptance  is  that  the  model-centric  process  is  not  worse,  not  less  reliable,  than  the 
current  process  in  any  area  or  aspect  of  airworthiness  and  safety  certification. 

It  is  our  opinion  that  these  identified  risks  are  potentially  manageable.  However  model  calibration  [78], 
validation  and  accreditation  for  MCE  with  respect  to  airworthiness  and  safety  may  require  significant 
effort  and  expertise  [47],  The  airworthiness  certification  handbook  (and  its  expanded  version),  and 
lessons  learned  from  previous  airworthiness  and  safety  assessments  provide  detailed,  but  incomplete, 
insight  into  the  resolution  and  fidelity  needed  in  the  models.  There  has  been  significant  progress  in  high- 
resolution  man-in-the-loop  simulators,  airworthiness  compliance  verification  via  simulation,  and  formal 
model  verification  and  completeness  processes. 


Future  Root  Causes 

As  the  focus  of  the  effort  is  on  understanding  the  problem,  including  pre-milestone  A  through  CDR,  it  will 
be  important  to  understand  MCE  approaches  to  assessing  the  potential  future  root  causes  of  risk 
especially  as  the  adversaries  are  attempting  to  leverage  unexpected  future  concerns,  for  example: 
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■  Adversaries  adapt  to  avoid  our  systems'  strengths  and  exploit  their  limitations  by  their  choice  of 
battlefields,  tactics,  and  equipment 

■  "Long-Lived"  DoD  Systems 

■  Systems  design  to  be  adapted  to  counter  adversary  adaptations  and  exploit  maturation  of  our 
emerging  technologies 

■  To  deter  and  defeat  current  threats 

■  To  enable  cost-effective  upgrade  &  adaptation 

This  is  not  an  exhaustive  list. 


Scope  of  the  Risk  Framework 

We  worked  with  our  NAVAIR  team  members  to  determine  the  scope  for  the  risk  framework.  Key  to  the 
representation  of  the  models  (and  Task  3)  to  support  risk  identification  and  management  is  to 
characterize  the  types  of  evidence  that  are  required  for  Flight  clearance  and  Flight  readiness.  It  is 
important  to  understand  how  the  models  are  developed  and  derived  in  order  to  understand  the  risk 
strategies  that  must  be  in  place  for  identifying  and  assessing  the  evidence  for  flight  clearance. 

The  process  for  risk  under  consideration  for  this  SE  transformation  covers  system  development  from 
Milestone  A  to  CDR  (at  least  for  now).  These  questions  related  to  risk  also  helped  to  refine  the  scope  for 
Task  3,  and  introduced  a  new  term  Digital  CDR  (DCDR),  with  a  heavy  emphasis  on  digitally-derived 
evidence  for  airworthiness  and  safety,  but  to  also  include  program  execution. 

In  both  preliminary  discussions  with  organizations  and  our  NAVAIR  team,  it  is  recognized  that  it  is 
important  to  quantify  "margins"  and  "sensitivities"  and  "uncertainties"  as  a  way  to  quantify  risk. 

As  an  example,  one  of  the  organizations  (in  our  preliminary  Task  1  discussion)  creates  new  types  of 
advanced  material  for  a  system.  They  cited  a  particular  effort  working  with  advances  in  new  material 
and  processes  at  the  nanoscale.  At  the  component  level  the  margins  seemed  acceptable.  However 
after  composing  the  components,  margins  propagated  to  unacceptable  levels  in  the  final  integrated 
form. 

Risk  implies  probabilities  of  what  might  go  wrong  or  might  not  happen  (on  time  or  due  to  the  degree 
expected),  and  some  distribution  of  magnitude  of  consequences.  This  requires  "impossible  certainty"  of 
the  degree  of  uncertainty  and  advance  knowledge  of  the  likelihood  and  effects  of  unidentified  events  and 
factors.  Therefore,  we  suggested  that  a  better  framework  might  be  to  work  in  terms  of  design  margin. 
Design  margin  is  more  closely  related  to  design.  Design  margin  is  how  much  room  there  is  for  a  subsystem 
or  component  to  perform  less  well  than  expected  or  to  have  greater  burdens  than  expected  until  it 
becomes  a  problem.  In  some  cases,  e.g.  weight,  any  increase  adds  to  total  weight,  so  instead  of  a  weight 
margin,  we  might  want  to  think  in  terms  of  sensitivities  (sensitivity  in  increase  in  total  weight,  time,  cost, 
etc.  to  a  percentage  increase  in  the  component  weight,  time,  power  draw,  etc.).  This  creates  a  number 
of  questions  for  this  task,  for  example: 

■  Can  we  use  models  to  see  how  much  design  margin  there  is  in  a  system  -  specifically  when  we 
cannot  push  the  system  to  failure;  which  types  of  models  and  how  can  we  use  them  to  estimate 
the  conditions  under  which  the  system  begins  to  exhibit  unstable  response 
o  In  control  systems  analysis  this  is  often  taken  to  be  the  3dB  point  -  the  frequency  of  input 
variation  at  which  the  output-to-input  ratio  is  half  what  it  was  for  low  frequency  change,  or 
the  90-degree  phase-shift  point,  where  the  frequency  of  input  variation  at  which  the  system 
response  lags  by  90  degrees 
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o  Control  systems  analysis  methods  also  address  the  acceleration,  velocity  and  displacement 
limits  at  which  the  system  dynamics  change 

o  Failures  are  often  associated  with  transitions  from  linear  to  highly  non-linear  regimes;  often 
the  structure,  interactions  and/or  dynamics  change  in  these  regions  (e.g.,  insulators  or 
isolators  fail,  etc.)  -  e.g.,  the  acceleration,  velocity  and  displacement  limits  at  which  the 
system  transitions  from  linear  to  non-linear  response 
o  Models  that  are  relevant  in  the  "linear"  regime  will  give  erroneous  results  in  the  non-linear 
regime 

o  Models  that  do  not  represent  the  dynamics  that  change  the  structure  of  a  system  (e.g., 
insulation  wearing  off  causing  a  short-circuit,  structural  failure  of  a  linkage,  strain  transitions 
from  elastic  to  plastic  deformation,  etc.)  will  give  erroneous  results 

Mechanical  or  electro-mechanical  control  and  isolation  systems  are  good  examples,  and  important  for 
airworthiness.  Control  systems  work  within  a  limited  range.  Standard  control  system  analysis  examines 
the  frequency  response  and  looks  for  the  3dB  frequency,  i.e.  the  frequency  at  which  the  transfer  function 
is  half  of  the  low-frequency  value  (the  transfer  function  is  just  the  ratio  of  output-to-input).  Other  limits 
include  maximum  displacement,  velocity  and  acceleration  -  when  the  system  hits  hard-stops,  current 
limits  etc. 

Surrogates  can  be  driven  with  increasing  frequency  inputs  to  find  the  3dB  point  without  having  to 
experience  the  failure.  The  input  parameters  of  virtual  models  are  often  "tuned"  to  match  the  3dB  point 
of  test  data,  and  then  used  to  extrapolate  to  find  the  3dB  point  of  hypothetical  systems.  Physically  realistic 
models  can  be  used  to  estimate  the  limiting  thresholds  of  stable  response,  provided  the  models  and 
inputs  are  adequately  calibrated  and  validated.  Special  consideration  is  needed  for  basic  physical 
processes  with  non-linear  response  in  the  regime  of  operation,  e.g.,  friction  between  moving  parts  versus 
friction  between  stationary  parts. 

Nested  control  loop  models  have  been  used  effectively  in  system  safety  modeling  and  analysis  [75],  The 
outer  control  loops  detect  changes  in  the  response  behavior  of  inner  control  loops,  and  then  adjust  the 
parameters  of  the  inner  control  loops  to  bring  the  inner  loops  back  into  the  stable  regime. 

In  the  use  of  modeling  and  simulation,  there  are  different  types  of  simulation  with  different  levels  of 
fidelity.  A  significant  challenge  is  that  tools  do  not  often  map  well  to  different  levels  of  abstractions.  These 
are  areas  to  frame  risk.  There  are  increasing  uses  of  model  transformation  from  one  level  or  to  different 
disciplines.  Model  transformation  and  model  consistency  between  these  views  becomes  a  risk  issue. 

A  companion  concept  is  credibility  of  the  estimates  of  performance,  cost,  etc.  High  credibility  if  it  has 
worked  in  a  surrogate  system,  less  if  it  is  similar  to  something  demonstrated  in  a  surrogate  and  model 
extrapolation.  It  will  be  important  to  better  understand  model  extrapolations. 

■  Less  credibility  the  farther  the  model  extrapolation  is  extended 

■  Less  credibility  going  from  surrogate  system  to  bench  testing,  etc. 

■  Use  of  multi-scale  calibration  and  validation 

■  Use  of  progressive  model-based  design  confirmation  in  technical  reviews 
o  Subsystems  mature  and  are  integrated  at  different  rates 

o  Sometimes  early  decisions  are  needed  for  long-lead  time  items  whose  specifications  can  be 
confirmed  before  other  aspects  of  the  system  (e.g.,  final  control  system  parameter  values) 
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Modeling  and  Methods  for  Uncertainty  Quantification 


Sandia  National  Laboratory  discussed  some  advanced  approaches  for  supporting  uncertainty 
quantification  (UQ)  to  enable  risk-informed  decision-making.  Their  methods  and  tooling  address  the 
subjects  of  margins,  sensitivities,  and  uncertainties.  The  information  they  provided  reflects  on  the 
advanced  nature  of  their  efforts  and  continuous  evolution  through  modeling  and  simulations  capabilities 
that  operate  on  some  of  the  most  powerful  high  performance  computing  (HPC)  resources  in  the  world. 
We  heard  about  their  HPC  capabilities,  methodologies  on  Quantification  of  Margins  and  Uncertainty 
(QMU),  an  enabling  framework  called  Dakota,  and  the  need  and  challenge  of  Model  Validation  and 
Simulation  Qualification  [106],  They  also  discussed  the  movement  towards  Common  Engineering 
Environment  that  makes  these  capabilities  pervasively  available  to  their  entire  engineering  team  (i.e.,  the 
designing  system  in  our  terminology).  We  think  their  capabilities  provide  substantial  evidence  for  the 
types  of  capabilities  that  should  be  part  of  the  risk  framework.  This  section  provides  additional  details. 


Dakota  Sensitivity  Analysis  and  Uncertainty  Quantification  (UQ) 

The  Dakota  framework  supports  optimization  and  uncertainty  analysis  [109],  There  is  significant  demand 
at  Sandia  for  risk-informed  decision-making  using  credible  modeling  and  simulation: 

■  Predictive  simulations:  verified,  validated  for  application  domain  of  interest 

■  Quantified  margins  and  uncertainties:  random  variability  effect  is  understood,  best  estimate 
with  uncertainty  prediction  for  decision-making 

■  Especially  important  to  respond  to  shift  from  test-based  to  modeling  and  simulation-based 
design  and  certification 

o  This  gets  to  an  important  point  about  how  to  use  models  as  opposed  to  testing,  which  is 
critical  for  NAVAIR's  objective  to  rapidly  and  continuously  "cross  the  virtual  V" 

The  HPC  capabilities  comes  into  play  as  they  are  built  to  take  advantage  of  the  HPC  environment  and  can 
be  combined  with  predictive  computational  models,  enabled  by  environment  and  culture  that  focuses  on 
theory  and  experimentation  to  help: 

■  Predict,  analyze  scenarios,  including  in  untestable  regimes 

■  Assess  risk  and  suitability 

■  Design  through  virtual  prototyping 

■  Generate  or  test  theories 

■  Guide  physical  experiments 

Dakota  is  referred  to  as  a  framework,  because  it  is  a  collection  of  algorithms  supporting  various  types  of 
integration  through  programmatic  (scripting)  interfaces;  this  is  representative  of  the  concept  of  model¬ 
centric  engineering,  see  Figure  29.  It  automates  typical  "parameter  variation"  studies  to  support  various 
advanced  methods  and  a  generic  interface  to  simulations/code,  enabling  QMU  and  design  with 
simulations  in  a  manner  analogous  to  experiment-based  physical  design/test  cycles  to: 

■  Enhances  understanding  of  risk  by  quantifying  margins  and  uncertainties 

■  Improves  products  through  simulation-based  design 

■  Assesses  simulation  credibility  through  verification  and  validation 

■  Answer  questions: 

o  Which  are  crucial  factors/parameters,  how  do  they  affect  key  metrics?  (sensitivity) 
o  How  safe,  reliable,  robust,  or  variable  is  my  system? 

(quantification  of  margins  and  uncertainty:  QMU,  UQ) 
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o  What  is  the  best  performing  design  or  control?  (optimization) 
o  What  models  and  parameters  best  match  experimental  data?  (calibration) 


General  Concept 


Example:  Electrical  Domain 
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Figure  29.  Dakota  Framework  Integration  Wraps  User  Application 


To  put  margins  and  uncertainty  into  context,  assume  that  there  is  a  device  that  is  subject  to  heat,  and  we 
need  assess  some  type  of  thermal  uncertainty  quantification.  Given  some  results  from  some  Design  of 
Experiment  (DoE)  (also  supported  by  Dakota)  results  that  give  a  probability  distribution  as  shown  in  Figure 
30  [2].  The  Mean  of  the  temperature:  T,  to  the  lower  bound  of  the  threshold  (e.g.,  72  degrees) 
characterizes  the  Margin,  and  the  Standard  Deviation  (T)  characterizes  the  uncertainty. 
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Figure  30.  Example  for  Understanding  Margins  and  Uncertainty 
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This  approach  and  Dakota  framework  supports  a  broad  set  of  domains,  and  therefore  we  think  it  can  be 
generally  applied  across  domain  for  NAVAIR,  for  example: 

■  Supports  simulation  areas  such  as:  mechanics,  structures,  shock,  fluids,  electrical,  radiation,  bio, 
chemistry,  climate,  infrastructure 

■  Is  best  used  with  a  goal-oriented  strategy: 

o  Find  best  performing  design,  scenario,  or  model  agreement 
o  Identify  system  designs  with  maximal  performance 
o  Determine  operational  settings  to  achieve  goals 
o  Minimize  cost  over  system  designs/operational  settings 
o  Identify  best/worst  case  scenarios 


69 


o  Calibration:  determine  parameter  values  that  maximize  agreement  between  simulation  and 
experiment 

■  Handles  parallelism,  which  is  often  not  feasible  with  commercial  tools,  and  why  HPC  can  play  an 
important  role 

■  Provides  sensitivity  analysis  -  find  the  most  influential  variables 

■  Uncertainty  Quantification 

o  Models  inherently  have  uncertainty 

o  Assess  effect  of  input  parameter  uncertainty  on  model  outputs 

•  Determine  mean  or  median  performance  of  a  system 

•  Assess  variability  in  model  response 

•  Find  probability  of  reaching  failure/success  criteria  (reliability) 

•  Assess  range/intervals  of  possible  outcomes 


An  Overview  of  Quantification  of  Margins  and  Uncertainty 

Dakota  is  a  tool  framework  that  can  support  the  method  of  Quantification  of  Margins  and  Uncertainty 
(QMU).  Some  of  the  material  from  Sandia  is  categorized  "Official  Use  Only  [OUO]."  We  provide  a  summary 
extracted  from  publically  available  information  [87], 

QMU  pre-dates  Dakota  and  is  not  unique  to  Sandia  as  it  was  used  at  Lawrence  Livermore  National 
Laboratory  and  Los  Alamos  National  Laboratory,  with  the  original  focus  of  the  methodology  to  support 
nuclear  stockpile  decision-making22.  QMU  is  a  physics  package  certification  methodology  and  although  it 
has  been  around  and  used  at  Sandia  dating  back  to  2003,  and  both  QMU  theory  and  implementation  are 
still  being  developed/evolved  [87],  We  believe  the  methodology  has  more  general  use  than  just  physics 
package  certification. 

QMU  applies  to  the  lifecycle  of  the  entire  weapon,  with  focus  on: 

■  Specification  of  performance  characteristics  and  their  thresholds 

o  Performance  is  the  ability  of  system/component  to  provide  the  proper  function  (e.g.,  timing, 
output,  response  to  different  environments)  when  exposed  to  the  sequence  of  design 
environments  and  inputs 

■  Identification  and  quantification  of  performance  margins 

o  A  performance  margin  is  the  difference  between  the  required  performance  of  a  system  and 
the  demonstrated  performance  of  a  system,  with  a  positive  margin  indicating  that  the 
expected  performance  exceeds  the  required  performance 

■  Quantification  of  uncertainty  in  the  performance  thresholds  and  the  performance  margins  as 
well  as  in  the  larger  framework  of  the  decisions  being  contemplated 

There  are  two  types  of  uncertainty  that  are  generally  discussed  that  account  for,  quantify,  and  aggregate 
within  QMU: 

■  Aleatory  uncertainty  (variability) 

o  Variability  in  manufacturing  processes,  material  composition,  test  conditions,  and 
environmental  factors,  which  lead  to  variability  in  component  or  system  performance 

■  Epistemic  uncertainty  (lack  of  knowledge) 

o  Models  form  uncertainty,  both  known  and  unknown  unknowns  in  scenarios,  and  limited  or 
poor-quality  physical  test  data 


22  The  Comprehensive  Nuclear  Test  Ban  Treaty  ends  full-scale  nuclear  weapons  testing  in  the  U.S.  President  Bill 
Clinton  at  the  United  Nations,  September  24,  1996 
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The  statistical  tolerance  interval  methodology  is  an  approach  to  quantification  of  margins  and 
uncertainties  for  physical  simulation  data.  There  is  also  probability  of  frequency  approach,  commonly 
used  in  computational  simulation  QMU  applications  [87],  which: 

■  Extends  the  "k-factor"  QMU  methodology  for  physical  simulation  data 

o  k-factor,  in  general,  is  defined  as  margin  divided  by  uncertainty  (M/U) 

•  Margin  (M):  difference  between  the  best  estimate  and  the  threshold  for  a  given  metric 

•  Uncertainty  (U):  the  range  of  potential  values  around  a  best  estimate  of  a  particular 
metric  or  threshold 

o  Provides  essential  engineering  analysis  to  ensure  the  collected  data  sample  includes 
measurements  that  may  be  used  to  infer  performance  in  actual  use 
o  It  is  important  to  understand  the  performance  requirement  to  understand  the  performance 
threshold  and  associated  uncertainty 

•  Threshold:  a  minimum  or  maximum  allowable  value  of  a  given  metric  set  by  the 
responsible  Laboratory 

■  The  new  method  addresses  the  situation  where  performance  characteristic  has  shown  the 
potential  for  low  margin  or  a  margin  is  changing  (likely  getting  smaller  or  there  is  greater 
uncertainty)  with  age  [87] 

o  Notionally  the  margin  shifts  from  the  mean  of  the  performance  characteristic  (PC)  and  its 
performance  requirement  (PR)  to  the  difference  between  a  meaningful  percentile  of  the 
distribution  of  the  performance  characteristic  and  its  performance  requirement 
o  Need  to  quantify  uncertainty  through  the  computation  of  a  statistical  confidence  bound  on 
the  best  estimate  of  the  chosen  percentile  rather  than  by  a  sample  standard  deviation  (as 
reflected  in  Figure  30),  which  does  not  account  for  sampling  variability 
o  This  is  accomplished  by  computing  a  statistical  tolerance  interval 

We  created  a  graphic  from  several  publically  available  sources,  as  shown  Figure  31  in  order  to  better 
explain  a  few  aspects  about  QMU,  Dakota,  epistemicand  aleatory  uncertainty.  Typically  within  the  Dakota 
framework  there  is  an  outer  loop:  epistemic  (interval)  variables  and  inner  loop:  uncertainty  quantification 
over  aleatory  (probability)  variables  (e.g.,  the  probability  distribution).  The  outer  loop  determines  interval 
on  statistics,  (e.g.,  mean,  variance).  The  inner  loop  uses  sampling  to  determine  the  responses  with  respect 
to  the  aleatory  variables.  This  information  can  be  used  to  understand  the  epistemic  and  aleatory 
uncertainties,  relative  to  the  Lower  Performance  Requirement  (LPR). 
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The  information  is  relevant  to  the  risk  framework  as  it  provides  evidence  about  methodologies  and  tools 
to  deal  with  several  of  the  topics.  QMU  and  Dakota  are  still  evolving,  and  there  are  a  number  of 
challenges: 

■  How  do  we  ensure  that  we  use  the  right  “data"  as  inputs? 

■  How  to  roll  up  to  the  system  level? 

■  Model  validation  and  simulation  qualification 


Risk  Framework  Approach  to  Uncertainty  Modeling  and  Prediction 

The  SERC  team  has  also  been  working  with  NAVSEA  to  develop  a  framework  and  approach  to  uncertainty 
quantification  modeling  and  prediction.  The  approach  has  three  main  components: 

■  Identifying  the  design,  test  and  modeling  factors  at  different  system  scales 

■  Analyzing  the  uncertainty,  variability,  and  error  in  design  implementation,  testing,  and  modeling 

■  Using  experimental  design  methods  to  assess  the  contributions  and  interactions  to  system 
(airworthiness  and  safety)  and  program  execution  risks 

The  risk  modeling  and  analysis  approach  also  addresses  potential  errors  and  uncertainties  in  the  overuse 
of  limited  data.  Ideally: 

■  One  data  set  is  used  to  identify  critical  factors 

■  A  second  independent  data  set  is  used  to  develop  the  models 
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■  A  third  independent  data  set  is  used  to  calibrate  the  models 

■  A  fourth  independent  data  set  is  used  to  assess  the  expected  error  in  model  results 

In  practice  data  sets,  surrogate  vehicle  test  data,  etc.  are  limited.  Bootstrap  methods  use  repeated 
resampling  of  the  data  and  repeating  the  modeling  and  analysis  process  to  obtain  a  statistical  estimate 
of  the  uncertainty  in  model-based  acquisition  given  the  available  data.  Further  analysis  reveals  the  value 
-  reduction  in  uncertainty  -  for  additional  data. 

These  types  of  models  capture  and  embed  knowledge  associated  with  expert  judgment,  historical 
evidence  and  rules  of  thumbs  that  are  used  in  the  decision-making  process.  Alternative  methods  help 
deal  with  these  type  of  issues. 


Predictive  Models  for  Risk 

There  are  situations  where  we  do  not  have  good  historical  quantitative  data  and  we  often  use  expert 
judgment.  This  section  discussions  a  predictive  modeling  approach  when  risk  involves  subjective 
information,  small  data  sets,  and  “dirty"  data. 

The  SERC  team  has  developed  and  used  models  in  the  prediction  of  risk,  and  plans  to  use  predictive 
analytic  models  to  support  risk  identification  and  management.  More  generally  we  can  use  models  to 
provide  risk  quantification  for  almost  all  types  of  decisions  that  are  made  by  stakeholders  (e.g.,  model- 
based  reviews).  As  an  example,  we  created  a  Bayesian  model  using  factors  derived  from  the  Airworthiness 
standard  MIL-HDBK-516B  [45]  as  shown  in  Figure  32.  This  is  conceptually  similar  to  the  approach  we  are 
using  on  an  FAA  NextGen  research  task  for  collaborative  risk-informed  decision-making  [15][16][17j.  The 
key  characteristics  of  the  approach  are  they  ensure  that  all  factors  are  considered  in  the  decision-making 
process,  and  that  all  classes  of  stakeholders  are  adequately  represented  in  the  decision-making  process. 
A  systematic  and  comprehensive  treatment  of  all  relevant  factors  provides  better  risk  identification. 

We  used  this  model  and  an  example  from  a  true  story  related  to  a  C130  Weapon  Delivery  system  to 
illustrate  the  concept.  While  this  model  is  notional  at  this  time,  this  example  started  a  discussion  with  the 
team  about  how  stochastic  (probabilistic)  models  can  play  an  important  part  of  the  Vision  as  they 
formalize  many  aspects  of  the  human  decision  making  process  that  will  be  important  at  many  gates, 
reviews,  and  decision  points  of  the  Vision  concept.  Each  factor  covers  a  specific  aspect  of  airworthiness, 
to  ensure  that  all  possible  uncertainties  and  risk  are  considered  in  the  quantification  of  risk.  The  risk  index 
is  a  probability  distribution,  where  for  example,  the  mean  can  map  to  quantities  in  a  risk  matrix. 
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Risk  Framework  Captures  Knowledge 

These  types  of  risk  frameworks  are  actually  knowledge  models  of  credibility  (not  models  of  performance, 
but  models  of  uncertainty).  Part  of  the  effort  on  modeling  the  "As  Is"  process  (Task  3)  is  to  identify  and 
then  formalize  within  the  models  the  information  and  associated  knowledge  for  evidence-based  decisions 
and  evidence-based  timing  of  decisions.  Other  considerations  and  opportunities: 

■  In  the  "As  Is"  process,  what  decisions  are  artifacts  of  the  process,  but  not  essential  to  the 
engineering  development? 

■  Are  there  lost  opportunities  by  making  early  concept  and  design  decisions? 

■  Is  there  a  risk  of  bad  decisions,  risks  and  costs  of  no  or  deferred  decisions,  during  planning,  or 
during  execution? 

■  Reconsider  the  "full  system"  technical  review  model.  Not  all  parts  of  the  system  are  ready  for 
PDR,  CDR  at  the  same  time.  Some  are  more  mature  than  others.  Maybe  a  granular  approach  is 
needed. 

The  timing  of  technical  reviews  and  decisions  should  be  made  when  there  is  an  accumulation  of  evidence 
sufficient  to  make  a  credible  decision.  Ideally,  this  will  be  inherent  in  the  Vision  concept,  when  the 
required  information  and  associated  analyses  are  complete,  the  evidence  and  timing  for  decisions  should 
be  triggered  events  in  the  automated  workflow. 
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Model  Validation  and  Simulation  Qualification 


Comparing  model  predictions  to  observed  responses  for  the  purpose  of  assessing  the  suitability  of  a 
particular  model  constitutes  what  is  known  as  model  validation.  Uncertainty  quantification  for  simulation 
models  is  not  strictly  limited  to  model  validation.  When  experimental  observations  are  available  for 
validation  assessment,  analysts  would  often  like  to  use  the  same  observations  for  model  calibration, 
which  is  the  process  of  adjusting  internal  model  parameters  in  order  to  improve  the  agreement  between 
the  model  predictions  and  observations.  However,  if  internal  model  parameters  can  be  adjusted  in  this 
manner,  this  means  that  there  is  some  amount  of  uncertainty  associated  with  the  true,  or  best,  values  of 
these  parameters.  Uncertainty  associated  with  model  inputs  directly  implies  uncertainty  associated  with 
model  outputs  [77], 

Model  validation  and  simulation  qualification  are  ways  to  ensure  that  "integrity"  of  the  models  prediction 
information.  Sandia  has  developed  the  "Real  Space"  model  validation  approach  [103],  which  was 
formulated  by  working  backwards  from  an  end  objective  of  "best  estimate  with  uncertainty"  (BEWU) 
modeling  and  prediction,  where  model  validation  is  defined  as:  the  process  of  determining  the  degree  to 
which  a  computer  model  is  an  accurate  representation  of  the  real  world  from  the  perspective  of  an 
intended  use  of  the  model.  However,  the  interpretational  and  implementation  details  can  still  vary 
widely. 

We  have  discussed  a  number  of  model  validation  and  simulation  qualification  topics,  such  as: 

■  Hierarchical  Model  Validation 

o  Seeks  to  expose  key  physics  and  material  models  that  are  brought  together,  and  asks  are 
the  combined  products  validated  at  various  levels  of  aggregation?  "right  for  the  right 
reasons" 

o  Seeks  to  catch  interactions  and  emergent  behaviors  not  present  in  validation  of  separate 
models 

o  Also  need  to  consider  "Traveling"  or  "Linking"  variables  that  bridge  modeling  levels  [106] 

■  "Exercising"  the  models  at  the  "boundaries"  of  the  probability  distributions  (~10  and  90 
percentile) 

o  This  is  related  to  a  recommended  testing  strategy  based  on  boundary-value  analysis/testing 
(i.e.,  exercising  the  "element  under  test"  at  the  boundaries  can  expose  more  anomalies  that 
exercising  the  nominal/typical  tests  scenarios) 
o  Has  greater  potential  to  expose  off-nominal  cases 

Various  model  validation  paradigms  and  methodologies  are  still  being  proposed,  developed,  and  tested. 
There  is  no  overriding  consensus  exists  yet  on  "best"  approach.  We  questioned  Sandia  about  an  idea  that 
we  had  in  our  working  session  about  how  we  are  increasing  in  the  ability  to  do  more  "integration"  of  the 
simulation  across  domains,  and  can  that  "integration"  provide  increased  visibility  into  potential 
anomalies,  therefore  allowing  us  to  better  understand  the  "integrity"  of  the  simulations.  This  is  analogous 
to  why  integration  testing  often  exposes  issues 

Sandia  provided  some  papers  that  we  can  share  with  the  team  [106],  This  information  provides  significant 
guidance  and  historical  perspectives  that  should  be  further  used  to  support  the  concept  of  model 
validation  and  model  integrity  as  part  the  Vision  for  Task  3.  In  addition,  Sandia  discussed  the  Predictive 
Capability  Maturity  Model  (PCMM),  which  is  an  evolving  model  that  can  be  used  to  assess  the  level  of 
maturity  of  computational  modeling  and  simulation  (M&S)  efforts  [88],  The  PCMM  addresses  six 
contributing  elements  to  M&S:  (1)  representation  and  geometric  fidelity,  (2)  physics  and  material  model 
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fidelity,  (3)  code  verification,  (4)  solution  verification,  (5)  model  validation,  and  (6)  uncertainty 
quantification  and  sensitivity  analysis. 

There  is  more  research  planned  for  a  follow-on  phase  of  this  research.  Here  are  other  topics  that  have 
discussed  related  to  improving  our  trust  in  models  and  simulation: 

■  Numerical  integration  techniques  [57] 

o  This  is  an  example  provide  by  NASA/JSC  related  to  simulation  of  space  vehicles  for  different 
planetary  bodies 

o  Propagating  the  evolution  of  a  vehicle's  translational  and/or  rotational  state  over  the  course 
of  a  simulation  is  an  essential  part  of  every  space-based  Trick  simulation.  The  underlying 
equations  of  motion  for  this  state  propagation  yield  second  order  initial  value  problems. 
While  analytic  solutions  do  exist  for  a  limited  set  of  such  problems,  the  complex  and 
unpredictable  nature  of  the  forces  and  torques  acting  on  a  space  vehicle  precludes  the  use 
of  analytic  methods  for  a  generic  solution  to  these  state  propagation  problems.  Numerical 
integration  techniques  must  be  used  to  solve  the  problem. 

■  Flights  validate  models/simulations 

■  Use  logged  data  to  continually  calibrate  models/simulation 

o  This  was  discussed  in  our  organizational  visits,  and  it  was  discussed  as  part  of  model 
guidance 

o  Model  calibration  should  be  getting  easier,  because  we  have  better  data  collection,  storage, 
and  the  ability  to  analyze  large  data  sets 

■  Models  of  pedigree 

■  Model  Validation  Review  (MVR) 

■  Cross-domain  integration  of  models  may  also  be  a  way  to  have  greater  confidence  in  simulation 

models 

o  We  know  that  integration  and  integration  testing  often  exposes  many  defects  or  anomalies 

o  We  currently  do  not  have  much  cross-domain  integration  of  models/simulation 

o  These  are  new  capabilities  and  the  inherent  nature  of  model-centricity  will  lead  to  greater 
integration;  this  could  potentially  provide  new  types  of  inputs/measures  (insights)  to  help  us 
build  trust  in  the  models 

■  Probabilistic  Risk  Analysis  -  this  might  be  yet  another  related  cross-domain  approach 

o  Organization  discussed  an  example  related  to  using  simulation  and  Dakota  to  reduce  the 
number  of  flight  tests 

■  Bayesian  model  calibration  [76] 

o  Model  calibration  is  a  particular  type  of  inverse  problem  in  which  one  is  interested  in  finding 
values  for  a  set  of  computer  model  inputs,  which  result  in  outputs  that  agree  well  with 
observed  data 

■  Bayesian-based  qualification  planning  [103] 

Finally,  Bill  Brickner  from  NAVAIR  points  out  that  no  mission-level  model  can  ever  be  validated  -  that  is, 
it  is  being  used  to  predict  possible  future  scenarios.  We  will  continue  to  investigate  approaches. 


Improving  the  Integrity  of  Models 

The  DARPA  META  program  discussed  the  issues  and  therefore  risks  with  attempting  to  use  models  that 
were  not  developed  according  to  a  methodology  that  results  in  models  that  are  suitable  to  the  capabilities 
of  the  tools  [7],  Increasingly,  there  are  approaches  being  used  to  improve  the  integrity  of  the  models 
through  various  types  of  checks,  for  examples: 
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■  NASA/JPL  use  an  ontology-based  approach  for  ensuring  models  are  developed  to  comply  with 
modeling  patterns,  and  have  developed  over  60,000  checks  using  computationally  based  formal 
reason  to  ensure  the  models  are  well-formed,  consistent  and  complete  with  respect  to  the 
modeling  pattern  [69] 

■  Formalized  evaluation  criteria  of  a  modeled  system  architecture  by  encoding  23  axioms  in  two 
different  tools:  Vitech  CORE  and  Innoslate  from  SPEC  Innovations  to  demonstrate  concept  [104] 
o  Some  tools  provide  pre-defined  checks,  but  in  this  case  the  tools  allow  for  additional  types 

of  checks  to  be  performed  that  can  be  adapted  to  a  particular  methodology 

■  Apply  an  approach  for  Verification,  Validation,  and  Accreditation  Shortfalls  for  Modeling  and 
Simulation  based  on  criteria  associated  with  modeling  and  simulation  patterns  in  SysML  [96] 

■  DARPA  META  also  developed  a  concept  of  Probabilistic  Certificate  of  Correctness  (PCC)  [7] 

o  PCC  is  used  to  verify  that  the  system  will  perform  within  the  requirements  bounds  within  a 
probabilistic  certainty  factor 

•  PCC  uses  statistical  sampling  techniques  on  system  and  environmental  parameters, 
executed  on  test  benches  for  evaluation  of  the  statistical  properties  of  the  metrics 

•  Requirements  objectives  and  thresholds  define  the  numerical  bounds  of  the  acceptable 
metric  values,  and  a  probability  of  the  design  metric  existing  within  the  bounds  is 
computed 

o  Variation  in  performance  is  expected  due  to  manufacturing  variations  of  components  and 
possibly  environmental  conditions 

A  risk  framework  needs  to  consider  these  and  other  types  of  checks.  These  types  of  checks  not  only  ensure 
that  the  model  is  well-formed,  consistent  and  complete  with  respect  to  modeling  patterns,  but  this  also 
reduces  risk,  because  these  types  of  rules  provide  greater  assurance  from  the  use  of  tools  that  use  these 
models  for  simulation,  analysis  and  generation. 


Model-Centric  Methods  and  Tools  Enable  Approaches  to  Safety  and  Airworthiness 

The  emerging  approach  to  Failure  Modes  and  Effects  Analysis  (FMEA)  and  safety  analysis  is  to  use  formal, 
automated  methods  to  derive  the  FMEA  and  safety  analysis  models  from  the  SE  model  of  the  system, 
instead  of  the  historical  practice  of  developing  informal  models  by  manual  means,  and  to  be  able  to 
integrate  failure  and  safety  analysis  earlier  in  design.  The  reliance  on  the  SE  model  of  the  system 
potentially  exposed  the  failure  mode  and  safety  analysis  to  the  risk  of  an  incomplete  or  incorrect  systems 
engineering  model.  This  risk  can  be  mitigated  by: 

1.  Using  formal  methods  based  on  a  meta-model  in  developing  the  system  model 

2.  Applying  automatic  model  verification  methods  to  the  system  model 

3.  Applying  error  and  uncertainty  propagation  methods  to  analyze  the  accuracy  of  the  system  model 

These  are  areas  of  active  research  and  development,  and  the  methods,  tools  and  techniques  are  rapidly 
maturing.  The  current  state-of-the-art  is  that  there  are  available  methods  with  demonstrated  capability, 
but  not  complete  capability  or  entirely  turnkey  operation.  Research  and  development  is  focusing  on 
integrating  software  FMEA  with  hardware  FMEA,  and  higher-level  resolution  of  failure  modes. 

We  believe  that  formalization  of  MCE  can  lead  to  advances  aligned  with  the  following  hypotheses: 

■  HI.  Model-based  methods  to  automatically  generate  Failure  Modes  and  Effects  Analysis  (FMEA) 
models  tools  (1)  are  feasible  and  practical,  (2)  are  superior  to  traditional  manual  methods,  (3) 
can  address  software  failure  modes,  and  software-hardware  interactions  in  cyber-physical 
systems. 
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■  H2:  Model-based  safety  analysis  methods,  procedures  and  tools  (1)  are  feasible  and  practical, 
and  (2)  are  suitable  for  analysis  of  mission  critical  requirements  and  design  in  the  phases  of  the 
acquisition  lifecycle  prior  to  flight  certification  testing. 

■  H3:  Risks  in  reliance  on  models  are  manageable,  and  model  analysis  methods  provide  a  level  of 
assurance  of  model  correctness  and  accuracy. 

We  provide  a  summary  of  some  research  finding  in  support  of  the  hypotheses. 


MCE  Methods  to  Generate  Failure  Modes  and  Effects  Analysis  (FMEA)  models 

Model-based  methods  to  automatically  or  semi-automatically  generate  Failure  Modes  and  Effects 
Analysis  (FMEA)  models  from  Systems  Engineering  (SE)  models  of  requirements,  function,  design  and 
their  interaction  are  feasible,  practical,  and  have  been  demonstrated.  There  are  alternative  technical 
approaches  to  enhancing  the  system  model  to  automatically  identify  abnormal  modes  and  chains  of 
effects.  The  automated  approaches  do  not  address  the  criticality  of  different  system  functions,  only  how 
the  functions  are  degraded.  The  automated  modeling  approach  provides  higher  resolution  of  degraded 
mode  states  for  high-level  functions  as  a  combinatorial  analysis  of  low-level  failure  states.  The  SE  models 
need  to  conform  to  a  meta-model  framework  for  automating  FMEA  models. 

Schindel  describes  a  systematic  approach  beginning  with  a  "metamodel"  to  organize  and  express  the 
system  requirements  and  high-level  design  MBSE  model  [110].  The  MBSE  model  defines  the  system 
architecture,  functional  requirements  and  functional  dependencies.  The  system  architecture  is  the 
organization  of  the  system  into  subsystems  down  to  a  "black  box"  level,  and  the  interconnections  or  links 
between  elements.  Functional  requirements  interaction  behaviors  and  performance  under  specified 
conditions  in  an  objective  technical  language  for  the  system  and  every  subsystem  down  to  the  "black¬ 
box"  resolution  of  the  MBSE  model,  and  for  each  of  the  links  between  elements.  Conditions  are  external 
and  internal  states  of  the  system  and  its  internal  subsystems.  Behaviors  are  state  changes.  Functional 
dependencies  are  the  hierarchical  relationships  among  functions,  in  which  higher-level  functions  depend 
on  lower  level  functions.  Requirements  are  automatically  converted  into  counter-statements  that  define 
the  failure  modes  -  failures  of  system  elements  and  links  to  perform  one  or  more  of  their  functions.  The 
effects  are  automatically  traced  through  the  links  and  nodes  in  the  system  architecture,  and  propagated 
up  through  the  functional  hierarchy. 

Increasing  complexity  makes  manual  methods  laborious,  and  highly  depends  on  the  experience  and 
expertise  of  the  practitioners.  Manual  approaches  are  subject  to  human  error  of  omission,  and  are  not 
tightly  linked  to  the  system  model.  These  issues  are  addressed  with  automated  FMEA  model  generation 
from  the  Systems  Engineering  model.  Automated  FMEA  model  generation  creates  the  possibility  to 
extend  FMEA  to  a  comprehensive  analysis  of  degraded  mode  states.  The  automated  methods  do  not 
distinguish  between  important  (likely  and  significant)  failure  modes  in  developing  the  FMEA  model  -  a 
process  guided  by  the  insight  and  experience  of  the  practitioners  in  manual  FMEA  modeling.  The  FMEA 
models  generated  by  automatic  methods  are  too  large  to  understand  by  reviewing  the  model.  Automated 
analysis  methods  are  needed.  These  methods  are  still  under  development.  Automated  analysis  requires 
input  data  regarding  the  likelihood  or  frequency  of  the  different  failure  modes,  and  the  importance  of 
high-level  function  degradation  mode.  Manual  models  require  the  same  data,  but  since  the  data  are  often 
not  available  (just  subjective  estimates),  the  dependencies  are  built  into  the  model  based  on  "expertise 
and  experience." 

Automated  FMEA  analysis  can  address  software  failure  modes,  and  software-hardware  interactions  in 
cyber-physical  systems.  FMEA  addresses  the  behavior  and  response  of  the  system  to  changes  in  (a) 
external  conditions  (environment,  demand),  and  internal  conditions  (performance  time,  accuracy  and 
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error  of  functional  processes  and  the  state  or  mode  of  logical  subsystems).  State  transition  modeling  is  a 
"best  practice"  to  specify  the  system  behavior  and  to  test  the  behavior  specifications.  Automated  FMEA 
for  software  requires  a  model  of  the  logical  states  and  transitions  of  the  information  in  the  system,  as 
well  as  the  physical  states  and  transitions  of  the  system  and  its  environment.  In  real-time  cyber-physical 
systems,  timing  of  effects  of  sensing,  processing,  signaling  and  response  on  transitions  needs  to  be 
included. 

There  are  many  possible  failure  causes  and  modes.  The  nominal  behavior  as  specified  in  the  state 
transition  model  may  not  be  the  desired  behavior,  i.e.  the  conceptual  design  may  not  satisfy  the 
requirements,  or  can  exhibit  unplanned  behaviors  ("sneak  circuits"  and  unplanned  feedback  loops).  The 
behavior  specifications  (the  state  transition  model)  may  be  incomplete,  i.e.,  there  are  possible  states  and 
transitions  that  are  not  represented  in  the  model,  e.g.,  the  effects  of  failure  of  an  actuator  (stuck  or 
random  response),  noise  or  failure  of  a  sensor,  short-circuits,  and  unexpected  combinations  of  logical 
information  states.  The  system  can  have  calibration  errors,  or  not  account  for  effects  of  temperature, 
pressure,  etc.  on  the  calibration.  Discretization  of  time  and  signal  level  can  lead  to  control  systems 
exhibiting  behavior  different  than  expected  from  a  continuous  model.  Software  specifications  can  be 
implemented  incorrectly,  i.e.,  coding  errors. 

Snooke  presents  an  approach  for  automated,  model-based  software  FMEA  [114].  He  addresses  three 
classes  of  software  failures:  abnormal  input  values  to  a  routine,  failure  of  the  processing  hardware,  and 
logical/algorithmic/semantic  error  in  the  software  code.  He  does  not  address  incomplete  or  incorrect 
specifications  for  the  software.  His  approach  combines  methods  from  software  diagnosis  (functional 
dependency  modeling)  with  automatic  fault  tree  methods. 

Another  approach  for  automated,  model-based  software  FMEA  demonstrates  the  method  in  an 
application  to  aircraft  flight  control  software  [126],  The  approach  is  concerned  with  failure  modes 
resulting  from  the  state  of  logical  information  in  the  system  and  the  transformation  of  information  state 
by  processing,  including  timing  constraints  and  synchronization.  The  approach  relies  on  a  system 
requirements  model  of  behaviors  and  performance  under  conditions,  and  a  software  requirements 
model  of  information  states,  state  transformations,  timing  interactions,  etc.  They  define  a  structured 
approach  to  system  and  software  modeling,  and  for  the  use  of  the  models  to  analyze  software  failure 
modes  and  effects.  They  applied  the  method  to  flight  control  software  that  controls  the  cabin  door,  the 
brake  system  and  the  front  flap. 


Mission-level  Model-based  Safety  Analysis 

Model-based  safety  analysis  methods,  procedures  and  tools  are  feasible  and  practical  for  mission  critical 
safety  assessment.  System  safety  analysis  methods  are  commonly  based  on  informal  system  models.  The 
lack  of  precise  models  of  the  system  architecture  and  its  failure  modes  often  forces  the  safety  analysts  to 
devote  much  of  their  effort  to  finding  undocumented  details  of  the  system  behavior  and  embedding  this 
information  in  the  safety  artifacts  such  as  the  fault  trees. 

NASA/JPL  developed  a  comprehensive  Safety-Driven,  Model-Based  System  Engineering  Methodology 
that  enables  system  engineers  to  design  systems  from  a  safety  point-of-view,  i.e.,  with  hazard  analysis 
folded  into  the  nominal  design  process  rather  than  conducted  as  a  separate  activity  [60],  This 
methodology  integrates  MIT's  Systems-Theoretic  Accident  Model  and  Processes  (STAMP),  STAMP-Based 
Hazard  Analysis  (STPA),  intent  specifications  (a  structured,  constraint-based  system  engineering 
specification  framework),  and  JPL's  State  Analysis  (a  model-based  systems  engineering  approach).  The 
methodology  was  developed  to  address  the  challenges  of  safety  design  and  analysis  for  complex, 
software-intensive  cyber-physical  systems,  and  to  do  so  early  in  the  design  process. 
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Model-based  safety  analysis  has  been  approached  as  a  special  case  of  automated,  model-based  FMEA 
approaches,  except  that  instead  of  addressing  all  system  requirements  and  their  derived  requirements, 
it  is  restricted  to  safety-related  requirements  and  their  derived  requirements.  Benefits  of  model-based 
safety  analysis  include: 

■  Tight  integration  between  systems  and  safety  analysis  based  on  shared  models  of  system 
architecture  and  failure  modes 

■  The  ability  to  simulate  the  behavior  of  system  architectures  early  in  the  development  process  to 
explore  potential  hazards 

■  The  ability  to  exhaustively  explore  all  possible  behaviors  of  a  system  architecture  with  respect  to 
some  safety  property  of  interest  using  automated  analysis  tools 

■  The  ability  to  automatically  generate  many  of  the  artifacts  that  are  manually  created  during  a 
traditional  safety  analysis  such  as  fault  trees  and  FMEA/FMECA  charts 

Joshi  and  Fleimdahl  leverage  existing  tools  and  techniques  from  model-based  development  to  create 
formal  safety  models  using  tools  that  are  familiar  to  engineers  and  the  static  analysis  infrastructure 
available  for  these  tools  [70],  They  enhance  the  system  model  with  a  fault  and  effects  model  to  automate 
much  of  safety  analysis,  and  do  so  in  a  way  that  the  safety  issues  point  back  to  design  elements  that 
produced  the  adverse  behavior.  Their  approach  composes  a  model  of  nominal  system  behavior  to  model 
of  failure  modes  and  effects  on  behavior  under  a  set  of  common  malfunctions  and  failure  modes  of 
elements  of  cyber-physical  systems.  Their  approach  uses  automated  model  checking  methods  that  will 
either  verify  safe  behavior  in  the  presence  of  "N"  faults,  or  find  a  combination  of  "N"  faults  that  produce 
adverse  behavior  (i.e.,  failure  to  meet  a  safety  critical  functional  requirement).  They  apply  the  method 
to  the  aircraft  wheel  braking  system  example  from  ARP4761  [108], 

Hi P-FHOPS  (Hierarchically  Performed  Hazard  Origin  and  Propagation  Studies)  is  an  automated  model- 
based  method  for  safety  analysis  that  enables  integrated  assessment  of  a  complex  system  from  the 
functional  level  through  to  the  low  level  of  component  failure  modes  [94],  The  failure  behavior  of 
components  in  the  model  is  analyzed  using  a  modification  of  classical  FMEA  called  Interface  Focused- 
FMEA  (IF-FMEA).  One  of  the  strong  points  of  this  approach  is  that  the  fault  tree  synthesis  algorithm  neatly 
captures  the  hierarchical  structure  of  the  system  in  the  fault  tree. 


Risk  Reliance  on  Model  Correctness  and  Accuracy 

Risks  in  reliance  on  models  are  manageable  and  model  analysis  methods  provide  a  level  of  assurance  of 
model  correctness  and  accuracy.  In  the  transformation  vision,  models  become  the  "language"  for  system 
development:  requirements  are  expressed  as  functional  models  in  SysML,  UML  or  similar  conventions, 
specifications  are  expressed  as  state-transition  models  (or  other  method  of  computation,  as  appropriate 
to  the  system),  the  specification  models  automatically  produce  code  and  component  performance 
specifications,  automatic  methods  generate  FMEA,  and  high-fidelity  system  operation  models  are  used 
for  virtual  testing  and  design  of  experiments  for  physical  testing. 

Using  incomplete  or  incorrect  models  at  any  stage  exposes  the  development  program  to  the  risk  of 
erroneous  design  decisions.  In  the  "As  Is"  document-centric  development  process,  the  program  has 
greater  exposure  to  this  risk  because  it  lacks  the  ability  to  test-and-verify  the  implications  of 
requirements,  specifications,  and  design,  and  because  the  opportunity  for  human  error  is  introduced  at 
each  state.  Model-centric  development  reduces  this  risk  as  a  metamodel,  as  shown  in  Figure  33  can 
formalize  the  required  information  and  relationships  need  to  ensure  a  complete  analysis.  As  a  cadre  of 
engineers  emerges  with  expertise  in  MCE  with  experience  using  improved  modeling  tools  and 
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frameworks,  and  model  checking  methods  and  tools,  the  risks  from  incomplete  or  incorrect  models  will 
decline. 


bdd  [package]  FMEA  [MetaModel]  J 


Figure  33.  Example  Failure  Modes  and  Effects  Analysis  Metamodel 

Using  formal  modeling  methods  based  on  a  meta-model  will  help  ensure  that  the  system  models  can  be 
checked  for  completeness  and  correctness  as  they  are  being  developed.  Automated  model  checking 
methods  procedures  and  tools  is  a  growing  field  with  commercial  products  and  expanding  scope.  These 
tools  automatically  check  model  completeness,  and,  to  some  extent,  correctness. 

Model-centric  development  introduces  a  different  risk- the  risk  of  uncritical  review  of  the  modeling  and 
analysis  methods  and  results.  In  document-centric  development  there  is  usually  health  skepticism,  and 
reliance  on  experienced  subject  matter  experts  to  review  the  development  documents.  In  model-centric 
development,  DoD  will  need  to  develop  a  cadre  of  experts  with  expertise  both  in  the  domain  and  in 
modeling  and  analysis  methods. 


Error  and  Uncertainty  Source  and  Propagation  Analysis 

Error  and  uncertainty  propagation  analysis  is  a  developing  area  of  research  and  development.  These 
methods,  procedures  and  tools  assess  the  accuracy  of  models,  and  put  "error  bars"  on  the  outputs.  This 
information  will  inform  decision  makers  with  a  realistic  confidence  in  model-based  decisions,  and  will 
inform  the  modelers  of  the  need  for  refined  modeling  and  calibration. 

The  goals  of  uncertainty  propagation  are  to  quantify  the  accuracy  of  model  outputs,  and  to  identify  the 
contributing  model  elements.  Many  methods  are  currently  in  use  or  being  researched  that  can  propagate 
error  through  a  system,  including  (1)  nondeterministic  analysis  via  brute  force  (such  as  Monte  Carlo  (MC)), 
(2)  univariate  dimension  reduction,  (3)  deterministic  model  composition,  (4)  error  budgets,  (5)  interval 


81 


analysis,  (6)  Bayesian  inference,  (7)  anti-optimizations,  and  (8)  error  propagation  via  Taylor-series 
expansion.  This  is  an  ongoing  area  of  research. 

NASA  Ames  identified  discretization  of  signals  and  discretization  of  time  as  an  source  of  error  in  cyber¬ 
physical  systems,  that  was  particularly  important  in  flight-critical  systems  since  the  effects  can  have  non- 
deterministic  effects  on  control  system  behavior.  Bhatt  et  al.  developed  a  model-based  analysis  method 
to  propagate  the  errors  associated  with  signal  type  and  range  bounds  through  the  model  to  analyze  the 
possible  effects  of  the  errors  on  the  cyber-physical  system's  behavior  [12].  They  demonstrate  the  run 
time  and  scalability  of  the  proposed  approach  on  a  set  of  avionics  models  developed  for  a  commercial 
aircraft. 

For  certification  requirements,  structural  loads  of  aircraft  have  to  be  demonstrated  with  required 
margins.  These  structural  loads  can  be  predicted  by  calculations.  Estimations  of  these  loads  can  only  be 
checked  and  confirmed  with  real  flight  measurements.  But  these  estimations  are  also  subjected  to 
uncertainties.  The  most  common  method  is  to  measure  the  effects  of  fight  loads  on  the  structure  during 
flight,  which  means  measuring  local  strains.  Another  method  is  by  measuring  pressure  distribution  during 
flight.  There  are  uncertainties  in  both  the  strain  (or  pressure)  measurements,  the  location  of  the  location 
of  the  measurement  points,  and  the  loads.  Gonzalez  et  al.  present  an  uncertainty  quantification  method 
for  flight  load  estimation  that  accounts  for  the  multiple  error  sources  [56], 


Risk  in  a  Collaborative  Environment 

Risk  is  not  limited  just  to  NAVAIR,  it  must  be  considered  during  the  interactions  with  contractors  in  a 
continuous  way  rather  than  the  monolithic  reviews,  especially  in  the  context  of  a  "radical 
transformation."  For  example,  can  we  create  a  means  to  enable  NAVAIR  to  continuously  use  model 
measures  as  an  assessment  of  the  design  and  risk  of  a  continuously  evolving  contractor's  design/system 
rather  than  having  document-based  reviews?  If  so,  then: 

■  There  might  be  a  need  for  new  types  of  policies  or  governance 

■  It  has  been  suggested  that  there  may  need  to  be  some  type  of  a  policy  reference  model  that: 
o  Provides  a  common  way  to  guide  the  use  of  artifacts  to  make  decisions 

o  Identify  evidence  (derived  from  models) 
o  Access  information 
o  Analyze  information  leading  to  a  Decision 

•  With  quantification  of  uncertainties 

■  Some  "radical"  transformation  thoughts 

o  A  continual  assessment  of  the  model  (all  of  the  models)  maturity 

•  Possibly  with  different  Model  Maturity  Levels  (MML) 
o  If  the  models  cover  all  aspects  of  the  aircraft 

•  Related  to  the  reference  architecture/model  of  an  aircraft 

o  We  can  have  a  cumulative  quantity  that  represents  the  state  of  the  design  and  a  measure  of 
risk  (uncertainty  relative  to  our  understanding  of  the  margins) 

It  must  be  stated  that  the  above  scenarios  about  "eliminating  reviews"  through  the  use  of  model 
measures  does  not  eliminate  the  interaction  between  NAVAIR  and  contractors,  rather  we  suggest  that 
there  is  a  need  for  continuous  collaboration  among  all  stakeholders  and  those  interactions  can  be  done 
on  a  weekly  basis  or  in  a  more  workshop-based  approach  in  the  context  of  models.  In  this  approach,  could 
a  "radical  transformation"  in  the  way  that  government  and  contractors  interact  reduce  risk?  One  of  the 
organizational  discussions  reflected  on  this  concept  of  continuous  collaboration  using  model;  the 


82 


following  is  a  true  and  positive  story  related  to  a  Navy  customer  in  the  use  of  Simulink  modeling  to 
overcome  numerous  issues: 

■  The  contractor  started  this  interaction,  because  they  were  several  years  behind,  and  had  not 
made  any  "real"  progress 

■  They  started  modeling,  which  uncovered  many  requirement  errors/issues 

■  This  helped  them  understand  the  complexity  and  realized  the  effort  was  much  more  extensive 
than  they  had  originally  estimated 

■  Started  open  discusses  with  their  customer  (Navy) 

o  Models  provided  tangible  technical  information  about  the  problem 
o  After  a  little  explanation  about  the  modeling  approach,  the  customer  was  able  to 
understand  the  models 

o  They  both  realized  that  requirement  shall  statements  cannot  provide  the  needed 

information,  and  many  were  just  wrong  (incorrect,  contradictory,  or  not  what  the  Navy 
wanted) 

o  Documented  issues  directly  in  the  models 

o  Realized  that  the  models  were  in  a  constantly  changing  state,  but  the  contractor  built  a  trust 
relationship  with  the  Navy  understanding  that  the  models  were  in  a  continuously  evolving 
state 

o  Each  passing  week  they  would  review  the  models  and  could  reflect  on  the  issues  that  were 
recorded  in  the  models 

There  are  other  variants  of  the  operational  model  that  were  recently  discussed  by  NASA/JPL  in  the  way 
that  they  use  models  and  reviews  in  a  different  way  than  the  traditional  "gate"  reviews  (e.g.,  SRR,  SFR)  in 
a  model-centric  way  [35], 


Risk  Related  Research 

SERC  research  teams  are  involved  in  several  related  research  efforts  that  will  be  leveraged  in  the  risk 
framework.  We  need  to  explore  how  the  following  can  be  leveraged: 

■  Trust  under  Uncertainty  -  Quantitative  Risk;  SERC  RT-107  [123], 

■  The  High  Performance  Computing  Modernization  (HPCM)  CREATE  [98]  program  to  use  high- 
fidelity  models  in  systems  design  is  establishing  a  working  group  on  Uncertainty  Quantification. 
SERC  partners  are  collaborating  with  NAVSEA  and  the  HPCM  program. 

■  The  DARPA  internet-fabrication  (i Fab)  project  sponsored  research  by  a  SERC  collaborator  to 
develop  software  to  automatically  detect  and  complete  gaps  in  specifications  for  a  "build  to" 
design. 

■  The  US  Army  TARDEC  is  developing  knowledge  models  to  capture  design  factors  and 
relationships  in  system  design  and  development.  The  resulting  decision  breakdown  structure 
and  process  should  help  distinguish  substantive  design  and  engineering  decisions  versus 
artifacts  of  the  "As  Is"  process.  SERC  partners  are  coordinating  with  this  effort  [52]  [117]. 

■  OSD  is  sponsoring  a  SERC  RT-142  Quantitative  Technical  Risk  project  to  identify  "risk  leading 
indicators"  and  "risk  estimating  relationships,"  analyzing  the  consistency,  completeness,  and 
complexity  of  the  system  architecture,  requirements,  task  structure,  and  team  organization,  and 
combining  those  with  TRL/IRL  levels  and  Advancement  Degree  of  Difficulty  indicators  (this 
project  is  being  conducted  in  collaboration  with  TARDEC  and  an  acquisition  program). 

■  The  Engineered  Resilient  Systems  (ERS)  [63]  effort  is  addressing  lost  opportunity  by  making  early 
concept  &  design  decisions,  the  time  and  cost  to  reverse  decisions,  and  tradeoffs  between 
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timely  but  bad  decisions  versus  deferred  decisions.  SERC  partners  are  collaborating  with  the 
NAVSEA  ERS  and  set-based  design  projects. 

Part  II  Summary 


Our  research  suggests  that  model-centric  engineering  is  in  use  and  adoption  seems  to  be  accelerating. 
Model-centric  engineering  can  be  characterized  as  an  overarching  digital  approach  for  integrating 
different  model  types  with  simulations,  surrogates,  systems  and  components  at  different  levels  of 
abstraction  and  fidelity  across  disciplines  throughout  the  lifecycle.  Organizations  are  progressing  beyond 
model-based  to  model-centric  where  integration  of  computational  capabilities,  models,  software, 
hardware,  platforms,  and  humans-in-the-loop  allows  us  to  assess  system  designs  using  dynamic  models 
and  surrogates  to  support  continuous  and  often  virtual  verification  and  validation  in  the  face  of  changing 
mission  needs. 

Enabling  digital  technologies  are  changing  how  organizations  are  conceptualizing,  architecting,  designing, 
developing,  producing,  and  sustaining  systems  and  systems  of  systems  (SoS).  There  are  many  enablers 
that  relate  to  characteristics  of  a  holistic  approach  (this  list  is  not  exhaustive): 

■  Mission-level  simulations  that  are  being  integrated  with  system  simulation,  digital  assets  & 
products  providing  a  new  world  of  services 

■  Leaders  are  embracing  change  and  adapting  to  use  digital  strategies  faster  than  others 

■  Modeling  environments  to  create  dynamic  operational  views  (e.g.,  DoDAF  OV-1)  are  increasingly 
used,  which  used  to  be  static  pictures 

■  ID,  2D  &  3D  models  have  simulation  and  analysis  capabilities  (mostly  physics-based)  are 
common  in  practice 

■  Platform-based  approaches  with  virtual  integration  help  automakers  deliver  vehicle  faster 

■  Modeling  and  simulation  in  the  automotive  domain  is  reducing  the  physical  crash  testing  (e.g., 
from  400  to  40);  this  could  imply  that  modeling  and  simulation  can  reduce  test  flights,  which  are 
very  costly  as  it  is  difficult  to  get  flight  clearances  on  air  craft  that  have  advanced  new 
capabilities 

■  Design  optimization  and  trade  study  analysis  allows  for  more  systematic  design  of  experiments 
and  allows  engineering  to  make  many  more  excursions  through  the  design  space 

■  Engineering  affordability  analysis  is  a  risk-based  approach  that  could  be  used  to  significantly 
reduce  flight  tests  by  focusing  on  those  flights  that  have  the  most  uncertainty  about  margins  of 
performance 

■  Risk  modeling  and  analysis 

■  Pattern-based  modeling  based  on  ontologies  with  model  transformation  and  analysis 

■  Domain-specific  modeling  languages 

■  Set-based  design 

■  Modeling  and  simulation  of  manufacturing 

This  list  is  not  exhaustive.  This  report  provides  scenarios  and  examples.  There  are  emerging  ideas  that  are 
envisioned  to  come  into  play  within  the  10-year  timeframe  of  NAVAIR's  future  state,  such  as: 

■  Computer  augmentation,  where  digital  assistance  will  begin  to  understand  what  we  are  trying  to 
model  and  through  advances  such  as  machine  learning  and  integrated  visualization  can  act  as  a 
knowledge  librarian  helping  us  to  model  some  aspects  of  the  problem  or  solution  at  an 
accelerating  pace 

■  Explosion  of  interactive  visualization,  which  we  will  need  as  we  have  a  "sea"  of  data  and 
information  derived  from  a  "sea"  of  models  with  HPC  computing  capabilities 
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Appendix  A:  Factor  Definitions 


The  following  is  the  current  set  (fifth  version)  of  the  set  of  factors  associated  with  the  discussion 
measurement  instrument.  As  the  discussions  with  organizations  are  held,  these  factors  and  the 
associated  categories  will  be  refined. 

Table  2.  Discussion  Instrument  Factor  Definition 


Factor  Category 

Factors 

General 

These  factors  relate  to  the  degree  to 
which  advance  MBSE  provides  a 
holistic  approach  to  SE 

Commentary 

Magnitude  of 
applicability 

over  the 

lifecycle 

Organizational  Scope 

What  is  the  scope  of  the  MBSE  usage? 
Normally,  when  thinking  about 
NAVAIR  systems  the  scope  is  quite 
large  and  involves  large  programs. 
Therefore,  what  organizational  scope 
does  the  MBSE  usages  apply:  Program, 
Project,  an  entire  Business  Unit, 
Platform  (e.g.,  a  type  of  a  specific 
aircraft,  tank,  automobile). 

Department,  or  Site. 

Key  to  all  of  these  questions  is  that  we 
are  looking  for  "Technical  Feasibility  of 
"doing  everything  with  model."  We 
recognize  that  actual  adoption  can  be 
difficult,  and  might  not  make  sense  on 
older  systems.  Therefore  related  to  this, 
it  is  probably  best  to  ensure  that  the 
question  perspectives  come  from  a 
Chief  Engineer,  Chief  Technical  Offer, 
MBSE  Organizational  Specialist  and 
possibly  the  Program  Manager.  To  carry 
this  a  step  further,  it  might  also  be 
important  to  keep  the  "systems" 
perspective  in  mind,  because  some  of 
the  concepts  discussed  may  have  been 
applied  in  Hardware  and  possibly  in 
Software  (e.g.,  the  control  laws  for  the 
F35  are  built  in  Simulink,  with  auto  code 
generation,  and  a  significant  portion  of 
auto  test  generation),  but  not 
completely  at  the  Systems  level. 

Scope  Impact 

How  broadly  does  the  answers  cover 
the  entire  lifecycle  (for  example,  a 
university  research  project  might  be 
very  advanced  in  terms  of  analysis  or 
simulation,  but  it  does  not  cover  the 
entire  DoD  5000  lifecycle). 

The  answer  to  this  question  has  a  lot  of 
weight,  because  we  need  to  consider 
answer  in  context  of  lifecycle  applicable 
to  NAVAIR  (and  in  general  DoD  Acq. 
Programs). 

Proven  beyond 
a  research 
concept 

Demonstrations 

Are  the  capabilities  discussed  actually 
in  operations  -  have  they  been 
demonstrated? 

We  want  to  understand  that  things 
discussed  are  more  than  just  research 
concepts. 

Crossing  the 
Virtual  V 

Integrated 

Simulation 

In  order  to  Cross  the  Virtual  V,  there 
will  be  many  types  of  modeling  and 
simulation  required  to  support  various 
type  of  domains  within  the  system.  To 
what  degree  are  the  simulations 
integrated,  and  better  yet  do  different 
simulations  work  off  of  shared 

models? 

In  order  to  "cross  the  virtual  V"  during 
the  early  stages  of  development,  it  is 
important  to  understand  if  the 
inputs/outputs  from  one  set  of 
simulations  can  feed  another  (e.g.,  to 
be  able  to  understand  the  capability  in 
the  mission  context) 

Formal  Analysis 

Are  the  analyses  (e.g.,  property 
analysis)  formal,  meaning  that  they 
are  performed  on  models 

automatically? 

Is  the  analysis  fully  automated  from  the 
models  (H)  or  is  there  human 
interpretation  required  (M  or  L)  or  none 
(L)? 
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Domain  Specific 

Are  the  different  types  of  models 
related  to  the  domain?  For  example, 
control  system  engineers  often  use 
Simulink/Matlab.  Also,  most  modeling 
and  simulation  environments  are 
domain-specific. 

Domain-specific  modeling  languages 
are  an  emerging  trend;  these  types  of 
approaches  provide  intuitive 

abstractions  (often  graphical)  that  are 
familiar  to  engineers  within  the 
domain.  Rather,  SysML,  while  good  for 
systems  engineers,  it  might  not  be 
applicable  to  flight  controls,  networks, 
fluid  dynamics,  etc.  In  addition,  there  is 
not  significant  support  for  automated 
V&V  from  SysML  as  the  semantics  are 
not  very  rich. 

Cross  Domain 
Coverage 

Domain 

Interoperability 

Are  the  models  that  are  in  different, 
but  related  domains  integrated?  Are 
the  models  consistent  across  the 

domains? 

For  example,  are  the  models  that  are 
used  for  performance  the  same  models 
used  for  integrity/dependability 

analysis? 

Synthesis/Generation 

Can  the  models  be  used  for 
synthesis/generation  of  other  related 
artifacts  such  as  code,  simulation, 
analysis,  tests  and  documentation 

Meta-Model/Model 

Transformations 

Are  the  models  used  in  one  domain,  or 
for  one  purpose,  transformable  into 
another  domain  where  the  well- 

defined  semantics  in  one  domain  is 
carried  through  the  transformation 
into  the  other  domain;  if  so  are  they 
known  to  be  consistent? 

We  know  that  one  type  of  modeling  is 
not  always  appropriate  for  everything, 
and  that  is  why  there  is  emergence  of 
Domain-Specific  Modeling  languages; 
the  key  question  is:  are  the  models  for 
one  use  consistent  for  other  users  (e.g., 
performance,  integrity). 

Virtual  System 
Representation 

Surrogate  Integration 

Are  surrogates  used  to  support 
analysis,  and  are  the  results  of  the 
surrogates  captured  so  that  they  can 
be  factored  into  modeling  and 
simulation  in  the  future? 

Example,  Formula  1  racing,  uses  data 
logging  during  physical 

experimentation  and  then  factors 
results  (and  logs)  back  into  simulation 
environment;  can  we  fly  some  new 
capability  on  an  existing  aircraft  and 
then  factor  the  results  from  the  test 
flights  back  into  the  modeling  and 
simulation  environments?  This  in  the 
future  should  allow  more  virtual  flight 
testing  (once  the  approach  becomes 
trusted). 

Formal  Capability 
Assessment 

How  well  do  the  models,  simulations 
and  analyses  capabilities  support  the 
ability  to  understand  the  capabilities 
being  developed? 

Are  the  abstractions  from  the  models 
still  "rich  enough"  to  be  representative 
of  the  actual  mission  environment 

when  used  in  a  virtual  environment? 
For  example,  if  we  use  a  3D  immersive 
environment,  can  we  understand  the 
physical  characteristic  of  the 

operational  system? 

Virtual 

Accuracy/Margin 

Analysis 

Are  the  modeling,  simulation  and 
analysis  accurate?  How  well  do  they 
allow  the  designers  to  understand  the 
margins? 

As  an  example,  margin  tolerances  at  the 
component  level  can  propagate  as  the 
system  is  composed  (or  assembled). 
Are  these  factors  understood  and 

controlled? 

3D  Immersive 

Environments 

What  is  the  degree  to  which  3D 
Immersive  Environments  are  used  to 
improve  the  understanding  (and 
possibly  training)  of  the  virtual 
systems. 
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Management 
Criticality  Risks 
(Relates  to  Task 

4) 

Risk  Management 

Is  there  proper  risk  management 
identification,  analysis  and  mitigations 
applied  based  on  the  use  of  models? 

This  should  also  consider: 

-  Adequately  deal  with  critical  timelines 

Integrated  operational  risk 

-  Change  management  (model-based 
change  management  is  different  than 
document-based) 

Predictive  Analytics 

Are  there  models  used  to  support  a 
quantitative  approach  to  risk 
management? 

Model-based  metrics 

Are  there  model-based  metrics  (or  a 
comprehensive  set  of  model 
measurements)  and  are  they  used  to 
support  the  management  of 
programs/projects? 

The  use  of  model-based  metrics  reflects 
on  the  modeling  maturing  of  the 
organization. 

Attributes  of 
Modeling 
Maturity 

Multi-model 
interdependencies  / 
consistency  and 
semantic  precision 

If  the  organization  is  dealing  with 
many  different  types  of  models,  are 
the  interdependencies  managed  and 
are  the  models  semantically  precise 
enough  to  manage  model 

consistency? 

Dealing  with  interdependencies  and 
modeling  consistency  often  deals  with 
having  a  detailed  understanding  of  the 
semantics  across  models.  Positive 
results  for  this  answer  suggest  a  very 
advanced  use  of  models. 

High  Performance 
Computing  (HPC) 

Is  HPC  applied  to  the  modeling, 
simulation  and  analysis  efforts? 

Use  of  HPC  is  an  indicator  of  high 
modeling  maturity. 

Procedures 

Are  the  procedures  for  using  the 
models  understood,  so  that  we  can 
trust  the  model  outputs  to  support 
other  related  types  of  analysis,  both  in 
terms  of  technical  as  well  as  risk? 

This  applies  heavily  in  airworthiness 
(e.g.,  MilStd.  516) 

Operational 
Risks  (Relates 
to  Task  4) 

Staff  and  Training 

With  the  advances  in  the  technologies 
associated  with  models,  are  the  staff 
and  training  in  place  to  support  the 
use  of  models? 

This  is  another  indicator  of  a  more 
advanced  organization.  As  a  side  effect 
the  use  of  3D  Immersive  systems  can  be 
valuable  in  both  collaboration  and  early 
training. 

Human  Factors 

How  well  are  human  factors 
supported  by  the  modeling,  simulation 
and  analysis  capabilities?  This  should 
consider  Usability. 

Certification 

How  well  do  the  models-based 
automation  and  practices  support 
certifications  (if  required)? 

If  not  applicable  use  M  -  for  Medium 

Indirect  support 
from  Models 

Regulation 

How  well  do  the  models-based 
automation  and  practices  support 
regulations  (if  required)? 

If  not  applicable  use  M  -  for  Medium 

Modeling  and 
Simulation 

Qualification 

How  much  do  we  trust  our  models? 
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Acronyms  and  Abbreviation 


This  section  provides  a  list  of  some  of  the  terms  used  throughout  the  paper.  The  model  lexicon  should 


have  all  of  these  terms  and 

many  others. 

AADL 

Architecture  Analysis  &  Design  Language 

ACAT 

Acquisition  Category 

AFT 

Architecture  Framework  Tool  of  NASA/JPL 

AGI 

Analytical  Graphics,  Inc. 

AGM 

Acquisition  Guidance  Model 

ANSI 

American  National  Standards  Institute 

AP233 

Application  Protocol  233 

ATL 

ATLAS  Transformation  Language 

ASR 

Alternative  System  Review 

AVSI 

Aerospace  Vehicle  Systems  Institute 

BDD 

SysML  Block  Definition  Diagram 

BN 

Bayesian  Network 

BNF 

Backus  Naur  Form 

BOM 

Bill  of  Material 

BPML 

Business  Process  Modeling  Language 

CAD 

Computer-Aided  Design 

CASE 

Computer-Aided  Software  Engineering 

CDR 

Critical  Design  Review 

CEO 

Chief  Executive  Officer 

CESUN 

International  Engineering  Systems  Symposium 

CMM 

Capability  Maturity  Model 

CMMI 

Capability  Maturity  Model  Integration 

COR  BA 

Common  Object  Requesting  Broker  Architecture 

CREATE 

Computational  Research  and  Engineering  for  Acquisition  Tools  and  Environments 

CWM 

Common  Warehouse  Metamodel 

dB 

Decibel 

DBMS 

Database  Management  System 

DAG 

Defense  Acquisition  Guidebook 

DARPA 

Defense  Advanced  Research  Project  Agency 

DAU 

Defense  Acquisition  University 

DCDR 

Digital  design  from  Critical  Design  Review  (CDR) 

DL 

Descriptive  Logic 

DoD 

Department  of  Defense 

DoDAF 

Department  of  Defense  Architectural  Framework 

DoE 

Design  of  Experiments 

DSL 

Domain  Specific  Languages 

DSM 

Domain  Specific  Modeling 

DSML 

Domain  Specific  Modeling  Language 

E/D  RAP 

Engineering  Data  Requirements  Agreement  Plan 

ERS 

Engineered  Resilient  Systems 

FAA 

Federal  Aviation  Administration 

FMEA 

Failure  Modes  and  Effects  Analysis 

FMI 

Functional  Mockup  Interface 

FMU 

Functional  Mockup  Unit 

88 


GAO 

HPC 

HPCM 

HW 

l&l 

IBM 

IBD 

ICD 

ICTB 

IDEFO 

IEEE 

INCOSE 

IPR 

IRL 

ISEF 

ISO 

IT 

IWC 

JEO 

JSF 

JPL 

Linux 

LOC 

M&S 

MARTE 

MATRIXx 

MBEE 

MBSE 

MBT 

MC/DC 

MCE 

MDA® 

MDD™ 

MDE 

MDSD 

MDSE 

MIC 

MMM 

MoDAF 

MOE 

MOF 

MOP 

MVS 

NASA 

NAVAIR 

NAVSEA 

NDA 

NDIA 


Government  Accounting  Office 

High  Performance  Computing 

High  Performance  Computing  Modernization 

Hardware 

Integration  and  Interoperability 
International  Business  Machines 
SysML  Internal  Block  Diagram 
Interface  Control  Document 
Integrated  Capability  Technical  Baseline 
learn  DEFinition  for  Function  Modeling 
Institute  of  Electrical  and  Electronics  Engineers 
International  Council  on  Systems  Engineering 
Integration  Problem  Report 
Integration  Readiness  Level 

Integrated  System  Engineering  Framework  developed  by  Army's  TARDEC 

International  Organization  for  Standardization 

Information  Technology 

Integrated  Warfighter  Capability 

Jupiter  Europa  Orbiter  project  at  NASA/JPL 

Joint  Strike  Fighter 

Jet  Propulsion  Laboratory  of  NASA 

An  operating  system  created  by  Linus  Torvalds 

Lines  of  Code 

Modeling  and  Simulation 

Modeling  and  Analysis  of  Real  Time  Embedded  systems 

Product  family  for  model-based  control  system  design  produced  by  National 

Instruments;  Similar  to  Simulink 

Model-based  Engineering  Environment 

Model-based  System  Engineering 

Model  Based  Testing 

Modified  Condition/Decision 

Model-centric  engineering 

Model  Driven  Architecture® 

Model  Driven  Development 
Model  Driven  Engineering 
Model  Driven  Software  Development 
Model  Driven  Software  Engineering 
Model  Integrated  Computing 
Modeling  Maturity  Model 

United  Kingdom  Ministry  of  Defence  Architectural  Framework 

Measure  of  Effectiveness 

Meta  Object  Facility 

Measure  of  Performance 

Multiple  Virtual  Storage 

National  Aeronautics  and  Space  Administration 

U.S.  Navy  Naval  Air  Systems  Command 

U.S.  Naval  Sea  Systems  Command 

Non-disclosure  Agreement 

National  Defense  Industrial  Association 
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NEAR 

Naval  Enterprise  Architecture  Repository 

NPS 

Naval  Postgraduate  School 

OCL 

Object  Constraint  Language 

OMG 

Object  Management  Group 

00 

Object  oriented 

OSD 

Office  of  the  Secretary  of  Defense 

OSLO 

Open  Services  for  Lifecycle  Collaboration 

0V1 

Operational  View  1  -  type  of  DoDAF  diagram 

OWL 

Web  Ontology  Language 

PDM 

Product  Data  Management 

PDR 

Preliminary  Design  Review 

PES 

Physical  Exchange  Specification 

PIA 

Proprietary  Information  Agreement 

PIM 

Platform  Independent  Model 

PLM 

Product  Lifecycle  Management 

POR 

Program  of  Record 

PRR 

Production  Readiness  Review 

PSM 

Platform  Specific  Model 

QMU 

Quantification  of  Margins  and  Uncertainty 

RT 

Research  Task 

RFP 

Request  for  Proposal 

ROI 

Return  On  Investment 

SAVI 

System  Architecture  Virtual  Integration 

SE 

System  Engineering 

SERC 

Systems  Engineering  Research  Center 

SETR 

System  Engineering  Technical  Review 

Simulink/Stateflow 

Product  family  for  model-based  control  system  produced  by  The  Mathworks 

SCR 

Software  Cost  Reduction 

SDD 

Software  Design  Document 

SE 

System  Engineering 

SFR 

System  Functional  Review 

SLOC 

Software  Lines  of  Code 

SME 

Subject  Matter  Expert 

SOAP 

A  protocol  for  exchanging  XML-based  messages  -  originally  stood  for  Simple  Object 
Access  Protocol 

SoS 

System  of  System 

Software  Factory 

Term  used  by  Microsoft 

SRR 

System  Requirements  Review 

SRS 

Software  Requirement  Specification 

STOVL 

Short  takeoff  and  vertical  landing 

SVR 

System  Verification  Review 

SW 

Software 

SysML 

System  Modeling  Language 

TARDEC 

US  Army  Tank  Automotive  Research 

TBD 

To  Be  Determined 

TRL 

Technology  Readiness  Level 

TRR 

Test  Readiness  Review 

UML 

Unified  Modeling  Language 

XMI 

XML  Metadata  Interchange 
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VxWorks 


XML 

US 

XSLT 

xUML 

Unix 

UQ 

VHDL 

V&V 


extensible  Markup  Language 
United  States 

extensible  Stylesheet  Language  family  (XSL)  Transformation 
Executable  UML 

An  operating  system  with  trademark  held  by  the  Open  Group 

Uncertainty  Quantification 

Verilog  Hardware  Description  Language 

Verification  and  Validation 

Operating  system  designed  for  embedded  systems  and  owned  by  WindRiver 


Trademarks 


Analysis  Server  is  a  registered  trademark  of  Phoenix  Integration,  Inc. 

Astah  SysML  is  Copyright  of  Change  Vision,  Inc. 

BridgePoint  is  a  registered  trademark  of  Mentor  Graphics. 

Cameo  Simulation  Toolkit  is  a  registered  trademark  of  No  Magic,  Inc. 

CORE  is  a  registered  trademark  of  Vitech  Corporation. 

IBM™  is  a  trademark  of  the  IBM  Corporation 
iGrafx  is  a  registered  trademark  of  iGrafx,  LCC. 

Java™  and  J2EE™  are  trademark  of  SUN  Microsystems 
Java  is  trademarked  by  Sun  Microsystems,  Inc. 

LDRA  is  a  registered  trademark  of  Trademark  of  LDRA  Ltd.  and  Subsidiaries. 

Linux  is  a  registered  trademark  of  Linux  Mark  Institute. 

Mathworks,  Simulink,  and  Stateflow  are  registered  trademarks  of  The  Mathworks,  Inc. 

MagicDraw  is  a  trademark  of  No  Magic,  Inc. 

MATRIXx  is  a  registered  trademark  of  National  Instruments. 

Microsoft®,  Windows®,  Windows  NT®,  Windows  Server®  and  Windows  VistaTM  are  either  registered 
trademarks  or  trademarks  of  Microsoft  Corporation  in  the  United  States  and/or  other  countries. 
ModelCenter,  is  a  registered  trademark  of  Phoenix  Integration,  Inc. 

Modelica®  is  a  registered  trademark  of  the  Modelica  Association. 

Object  Management  Group  (OMG):  OMG's  Registered  Trademarks  include:  MDA®,  Model  Driven 
Architecture®,  UML®,  CORBA®,  CORBA  Academy®,  XMI® 

OMG's  Trademarks  include,  CWM™,  Model  Based  Application  Development™,  MDD™,  Model  Based 
Development™,  Model  Based  Management™,  Model  Based  Programming™,  Model  Driven  Application 
Development™,  Model  Driven  Development™ 

Model  Driven  Programming™,  Model  Driven  Systems™,  OMG  Interface  Definition  Language  (IDL)™, 
Unified  Modeling  Language™,  «UML»™ 

OMG®,  MDA®,  UML®,  MOF®,  XMI®,  SysML™,  BPML™  are  registered  trademarks  or  trademarks  of  the 
Object  Management  Group. 

Oracle  and  Java  are  registered  trademarks  of  Oracle,  Inc.  and/or  its  affiliates. 

ParaMagic  is  a  registered  trademark  of  InterCAX,  Inc. 

PHX  ModelCenter  is  a  registered  trademark  of  Phoenix  Integration,  Inc. 

PowerPoint  is  a  registered  trademark  of  Microsoft,  Inc. 

Real-time  Studio  Professional  is  a  registered  trademark  of  ARTiSAN  Software  Tools,  Inc. 

Rhapsody  is  a  registered  trademark  of  Telelogic/IBM. 

Rose  XDE  is  a  registered  trademark  of  IBM. 

SCADE  is  copyrighted  to  Esterel  Technologies. 
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Simulink  is  a  registered  trademark  of  The  MathWorks. 

Stateflow  is  a  registered  trademark  of  The  MathWorks. 

Statemate  is  a  registered  trademark  of  Telelogic/IBM. 

STK  is  a  registered  trademark  of  Analytical  Graphics,  Incorporated  (AGI),  Inc. 

UNIX  is  a  registered  trademark  of  The  Open  Group. 

VAPS  is  registered  at  eNGENUITY  Technologies. 

VectorCAST  is  a  registered  trademark  of  Vector  Software,  Inc. 

Visio  is  a  registered  trademark  of  Microsoft,  Inc. 

VxWorks  is  a  registered  trademark  of  Wind  River  Systems,  Inc. 

Windows  is  a  registered  trademark  of  Microsoft  Corporation  in  the  United  States  and  other  countries. 
XML™  is  a  trademark  of  W3C 

All  other  trademarks  belong  to  their  respective  organizations. 
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